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CHAPTER  1 


INTRODUCTION 


OVERVIEW 

Three  weeks  after  the  Japanese  attack  on  Pearl 
Harbor,  the  Army  Air  Force  (AAF)  only  had  9,000  trained 
pilots;  in  less  than  three  years  the  Flying  Training 
Command  trained  more  than  190,000  pilots.  It  was  during 
this  period  of  urgent  growth  that  the  first  widespread 
use  of  aircraft  simulators  appeared.  The  crude  Link 
Trainers  of  WW  II  started  the  Air  Force  on  the  road  to 
the  multimillion  dollar  mission  simulators  of  today 
(13:73-75) . 

The  use  of  aircrew  training  devices  (ATDs)  — 
modern  versions  of  the  WW  II  Link  trainer  —  increased 
along  with  the  cost  of  modern  aircraft  and  aviation. 
Strong  incentive  for  increased  ATD  use  came  from  1973 
Congressional  pressure  to  reduce  military  flying  hours  by 
25%  (12:5-1).  The  goal  of  simulator  builders  became  to 
reproduce  the  flying  environment  so  precisely  as  to  re¬ 
place  actual  flying  hours  with  simulator  hours. 

[Simulator]  effectiveness  is  directly  and  propor¬ 
tionally  related  to  the  realism  produced  by  the 
training  system  in  the  sense  that  the  simulator 
experience  must  transfer  to  the  ultimate  goal  of 
combat  effectiveness  [23:54]. 
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As  ATD's  realism  increased,  costs  skyrocketed  to  such 
heights  as  to  make  large  volume  buys  of  ATDs  impossible. 
Today  accessibility  to  the  limited  number  of  ATDs  is  a  new 
problem. 

It  does  the  Air  Force  little  good  to  own  the  world's 
most  realistic  and  sophisticated  simulator  if  there  are 
not  enough  of  them  to  provide  all  of  its  crews  with 
sufficiently  frequent  access  [23:54], 

Can  the  Air  Force  continue  to  provide  effective  training 

for  new  and  more  costly  aircraft  while  maintaining  ATD  costs 

at  levels  under  that  of  the  "realistic"  ATDs  of  today?  If 

so,  how? 

PROBLEM  STATEMENT 

An  ATD's  configuration  is  ultimately  driven  by  the 
weapon  system's  performance  and  physical  characteristics, 
as  well  as  user  maintenance,  training,  and  operational  con¬ 
cepts.  These  data,  along  with  other  supporting  weapon  sys¬ 
tem  (W/S)  program  and  contractor  engineering  data,  are  used 
by  the  training  developer  to  develop  training  objectives, 
the  base  line  against  which  the  evolving  ATD  is  developed 
and  tested  (43:3).  In  an  ATD  program  supporting  a  weapon 
system  under  development,  the  data  upon  which  the  training 
analysis  hinges  are  constantly  changing  and  typically  are 
not  frozen  until  the  weapon  system's  ultimate  configuration 
is  determined.  Given  this  acquisition  environment,  this 
thesis  studied  two  problems  in  the  ATD  development  process: 
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1.  ATD  acquisition  and  training  personnel  needed  a 
better  way  to  develop  training  and  ATD  requirements  for 
developing  weapon  systems . 

2.  ATD  acquisition  personnel  needed  a  better  pro¬ 
cess  to  transform  ATD  requirements  into  performance  speci¬ 
fications. 

The  following  highlighted  the  need  to  study  this 

area: 

1.  Delivered  ATD  systems  were  ineffective  leading 
to  increased  flying  time  in  order  to  meet  required  profi¬ 
ciency  levels  (34:3). 

2.  ATDs  were  rarely  delivered  prior  to  the  weapon 
system's  Initial  Operational  Capability  (IOC)  date  as  re¬ 
quired  by  regulations  (12:5-2)  . 

3.  Engineers  were  forced  to  work  with  poorly  iden¬ 
tified  requirements  (21)  leading  to  system  specifications 
that  did  not  satisfy  training  requirements.  In  addition, 
without  adequate  direction,  the  engineer  tried  to  meet  all 
specifications  totally  resulting  in  an  ATD  which  is  exces¬ 
sively  expensive  and  has  excess  capabilities  ( "goldplated") 
(54)  . 

4.  At  an  ATC/AFSC/HQ  USAF  workshop  on  acquisition 
and  Instructional  Systems  Development  (ISD)  (the  mandated 
method  for  analysis  of  training  requirements) ,  ATC  made 
the  following  statement: 
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.  .  .  the  methods  developed  by  ATC  in  1973-77  to 
permit  ISD  application  no  longer  meet  system  acqui¬ 
sition  schedules.  In  particular,  training  equipment 
identification  is  being  required  by  system  program 
offices  before  sufficient  data  exists  to  permit  ISD 
analysis.  ATC  is  uncertain  how  to  proceed  [25:Atch.2]. 

5.  In  1980,  at  a  workshop  on  the  Front  End  Analy¬ 
sis  (FEA)  problem,  one  of  the  participants  voiced  the 
following  concerning  ATD  requirements  for  weapon  systems 
under  development: 

With  the  occurrence  of  frequent  changes  in  equip¬ 
ment  design,  how  can  we  get  data  on  actual  equipment 
to  simulator  designers  so  that  appropriately  designed 
training  devices  and  simulators  are  developed  and 
fielded  in  a  timely  manner  [34:3]? 

This  sample  of  the  problem's  effect  on  the  acqui¬ 
sition  of  training  systems,  the  rising  cost  of  simulation 
systems  for  training  (23:56,59),  and  comments  by  the 
current  Deputy  Secretary  of  Defense,  Frank  C.  Carlucci, 
that  the  DOD  should  examine  the  acquisition  process  for 
improvements  provided  the  initial  motivations  for  this 
thesis  (2:9) . 

BACKGROUND 

Four  background  subjects  relevant  to  the  thesis  are 
reviewed  in  this  section.  First,  the  current  conceptual 
framework  used  to  develop  ATD  training  requirements,  ISD, 
is  discussed.  Second,  the  acquisition  setting,  called  the 
weapon  system  acquisition  process,  is  reviewed.  Third,  the 
structured  programming  concept  is  discussed  in  general  as 
a  method  of  viewing  complex  systems.  Fourth,  the  current 
method  Aeronautical  Systems  Division  (ASD)  uses  to  procure 
ATDs  is  reviewed. 
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INSTRUCTIONAL  SYSTEMS  DEVELOPMENT.  Air  Force 


Regulation  (AFR)  50-8  requires  that  all  Air  Force  training 
and  education  programs  are  developed  and  conducted  in 
accordance  with  the  ISD  process. 

ISD  is  defined: 

ISD  is  a  systematic  but  flexible  process  used  to 
plan,  develop,  and  manage  education  and  training  pro¬ 
grams.  When  used  properly,  the  ISD  process  helps  man¬ 
agers  plan  and  use  training  resources  effectively. 

The  ISD  process  identifies  training  requirements; 
translates  those  requirements  into  valid  learning 
objectives;  selects  the  proper  training  strategy; 
develops  effective  training  delivery  systems;  and 
provides  quality  control.  Using  ISD  makes  sure  that 
people  get  the  knowledge,  skills,  and  attitudes  needed 
to  do  their  Air  Force  jobs  [45  :lj  . 

The  ISD  process  is  further  defined  as  the  five  step  process 
shown  in  Figure  1-1.  ISD  starts  with  an  analysis  of  the 
job  (Step  1)  and  proceeds  through  the  remaining  steps.  Of 
particular  interest  is  Step  4  which  includes  the  first 
point  where  training  media  —  including  ATDs  —  are  iden¬ 
tified.  The  validity  of  the  process  is  then  reviewed. 
(20:39-53;  45 :l-2  to  1-3) . 

ISD-type  processes  are  validated  by  their  success 
in  military  and  industry  applications.  Mager  Asso¬ 
ciates,  Inc.  developed  and  marketed  an  ISD-type  process 
called  Criterion-Referenced  Instruction  (CRI)  with  success. 
A  partial  list  of  organizations  using  CRI  (Figure  1-2) 
include  both  civilian  and  military  organizations  with  large 
and  diverse  training  tasks  (24:6).  In  addition,  Mager' s 
promotional  literature  includes  customer  comments  such  as: 
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CONSTRAINTS 


del  for  ISD  (45«Figure 


Xerox 

Rank  Xeros  (UK) 

Gulf  Oil  Canada 

Cii  Honeywell  Bull  (France) 

Bell  Telephone  Co. 

U.S.  Air  Force 
Scandinavian  Airlines 
Air  Canada 

South  African  Broadcasting  Corp. 
Crocker  Bank 
U.S.  Army 

Hongkong  &  Shanghai  Banking  Corp. 
Continental  Telephone  Service  Corp. 
Technicon 
United  Telecom 

Royal  Canadian  Mounted  Police 
Pet  Foods,  U.K. 

Mitsui  Machinery  Sales  (UK)  Ltd. 
Center  for  Disease  Control 


Figure  1-2.  Some  Organizations  Currently  Using  CRI 
Materials  and  Procedures  (24:6). 


I  have  found  the  application  of  the  Criterion 
Referenced  Instruction  to  have  these  benefits: 

•Makes  the  instructional  developer's  job  easier... 
•Reduces  development  costs... 

These  two  benefits  have  had  a  recognizable  impact 
on  our  ability  to  deliver  a  higher  quality  training 
program  to  a  client  division  in  less  time  than  we  have 
experienced  in  the  past. 

Arlan  L.  Tietel 

Learning  Systems  Supervisor 

Education,  Training  and  Development  Department 
3M  Company  [24:5]. 

The  ISD  process  establishes  the  environment  in  which  train 
ing  developers  must  operate. 
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Systematic  instruction  in  the  Air  Force  began  in 
1970  and  has  evolved  into  a  current  set  of  ISD  documenta¬ 
tion.  In  1970,  the  Vice  Chief  of  Staff,  USAF,  directed  a 
"systems  approach"  to  military  training  (5:396).  That 
directive  evolved  into  current  Air  Force  directives: 

1.  Air  Force  Regulation  (AFR)  50-8:  States  the 
requirement  to  use  ISD  in  all  training  related  matters  and 
assigns  responsibilities  (45:i-4). 

2.  Air  Force  Manual  (AFM)  50-2:  Summarizes  the 
ISD  process  (44 :i) . 

3.  Air  Force  Pamphlet  (AFP)  50-58,  Volumes  I  -  VI : 
States  the  "how-to"  of  each  ISD  step  and  sub-step 

(42 : Vol . I , i) . 

4.  Others:  Other  directives  dictate  ISD's  use  in 
training  related  activity.  Of  special  interest  is 

AFR  50-11  which  regulates  ATD  development,  acquisition,  and 
use.  AFR  50-11  states  that  ISD  must  be  used  for  ATD  require¬ 
ments  process  and  certification  (46:1,8,9). 

The  acquisition  process  is  discussed  in  several  of 
the  above  publications.  AFR  50-8  states  that  Air  Force 
Systems  Command  (AFSC)  has  primary  responsibility  for 
acquiring  ATD's  concurrently  with  new  weapon  system  acqui¬ 
sition  (45  :2-3) .  AFM  50-2  warns  the  reader  of  possible 
ISD/acquisition  problems: 
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All  the  parts  of  a  complete  weapon/support  system 
are  usually  not  in  the  same  stage  of  development  at 
the  same  time.  Thus,  while  there  may  be  much  infor¬ 
mation  available  on  some  subsystems,  there  may  be 
other  subsystems  developing  more  slowly  for  which 
very  little  information  is  available.  This  is  a 
problem  in  designing  the  instructional  system  since 
you  may  have  to  make  instructional  planning  decisions 
before  all  of  the  information  is  available.  To  the 
extent  that  data  have  been  gathered  and  analyzed  in 
applying  the  ISD  process  at  any  given  time,  you  have 
a  more  solid  basis  for  making  instructional  planning 
decisions.  As  the  weapon/ support  system  develops  and 
more  of  the  design  becomes  firm,  more  information 
will  be  available.  Use  this  to  update  the  instruc¬ 
tional  planning  decisions  [44:2-3]. 


And : 


If  you  fail  to  identify  these  items  [with  long 
lead-times  like  ATDs]  until  course  development  is  com¬ 
plete,  that  will  hurt  the  ability  of  the  instructional 
system  to  provide  qualified  personnel  in  a  timely 
manner  [44 : 3-6]. 

The  ISD  process  provides  the  framework  within  which 
current  ATD  acquisition  takes  place,  and  the  ISD/ATD  pro¬ 
cesses  takes  place  in  the  larger  weapon  system  acquisition 
process. 


WEAPON  SYSTEM  ACQUISITION.  The  system  concept  goal 
is  to  design,  produce,  and  manage  the  weapon  system  as  an 
integrated  unit  instead  of  a  series  of  individual  efforts. 
Therefore,  the  total  weapon  system  includes  several  sub¬ 
systems  such  as  the: 

1.  aerospace  vehicle, 

2.  support  equipment, 

3.  facilities, 


4.  technical  data, 


5 .  supply , 

6.  maintenance, 

7.  personnel, 

8 .  training. 

All  subsystems  are  developed  concurrently  with  the  basic 
airframe  with  the  goal  of  having  all  subsystems  ready  when 
needed  to  support  the  aircraft.  The  training  system,  along 
with  its  ATDs ,  is  one  of  a  major  weapon  system's  subsys¬ 
tems  (11:30-33;  26:3).  A  five  phase  process  is  used  to 
take  the  entire  weapon  system  from  conceptualization  to 
deployment . 

The  five  phase  system  acquisition  process  is 
"essentially  a  logical  flow  of  activity  representing  an 
orderly  progression  from  concept  formation  to  final  opera¬ 
tional  deployment  [26:6]."  An  overview  of  each  phase  is 
found  in  Figure  1-3  (11:44).  Each  successive  phase  is 
designed  to  develop  increased  confidence,  to  reduce  risks 
in  the  acquisition,  and  to  provide  incremental  commitments 
of  resources  to  the  project.  In  addition,  decision  mile¬ 
stones  provide  information  about  and  direction  to  system 
decision  makers.  And  as  discussed  by  Gladney  and  Jeas, 
the  amount  of  information  about  the  system  increases  as 
the  acquisition  process  moves  through  each  phase  (15:81). 
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OPERATIONAL  EXPERIENCE 


SUPPORTING  TECHNOLOGY 


STRUCTURED  ANALYSIS.  Very  quickly  in  our  review  of 
the  ATD  acquisition  problem,  similarities  became  apparent 
between  what  ISD  people  needed  and  computer  programming 
people  had.  While  ISD  provided  training  developers  with  a 
methodical  way  to  analyze  existing  weapon  systems  (starting 
with  the  task  analysis) ,  they  needed  a  satisfactory  tool  for 
weapon  systems  under  development.  Computer  programmers  had 
developed  an  approach  called  Structured  Analysis  (SA)  to 
"attack"  large,  complex  problems.  This  approach  may  be  use¬ 
ful  in  developing  training  systems  and  ATDs . 

Since  SA  is  essentially  a  systems  approach,  Victor 
Weinberg  (52)  built  on  several  systems-related  terms.  SA 
is  the  analysis  of  systems  which  Weinberg  described  in  the 
following  way:  he  said  that  to  analyze  a  system  is  to 

...identify  the  complex,  its  components  (people, 
machines,  rules,  and  procedures) ,  and  their  interre¬ 
lationships  to  determine  objectives,  requirements, 
priorities,  and  the  extent  to  which  they  all  have  been 
satisfied  [52:10]. 

Weinberg  then  provides  two  definitions  of  systems  analysis. 
The  definition  that  relates  best  to  our  research  is  the 
following: 

...the  examination  of  problems,  objectives,  re¬ 
quirements,  priorities,  and  constraints  in  an  environ¬ 
ment,  plus  identification  of  cost  estimates,  benefits, 
and  time  requirements  for  tentative  solutions  [52:13]. 

Figure  1-4  is  a  graphical  presentation  of  this  definition. 

The  definition  of  structured  analysis  follows  from 
the  definition  of  systems  analysis.  SA  is  "a  disciplined 
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approach  to  structuring  the  systems  analyst's  job  [52:33]. 
In  addition,  it  is  "defined  as  a  philosophical,  top-down 


approach  to  all  phases  of  the  systems  life  cycle  [52:33].” 


Figure  1-4.  System*  Analysis  Diagram  (52:Figure  1-3). 


From  the  definition  of  SA,  Weinberg  details  "impor¬ 
tant  guidelines"  for  implementation  of  SA: 

1 .  Get  the  users  to  participate  in  the  development 
and  evaluation  of  systems. 

2.  Consider  the  technical  level  of  expertise  and 
the  bottom-line  objectives  of  the  users  when  producing 
documents  for  user  review. 

3.  Use  graphic  tools  to  minimize  potential  com¬ 
munication  problems. 

4.  Build  a  logical  systems  model  before  concen¬ 
trating  on  detailed  physical  requirements. 

5.  Take  a  disciplined  top-down  approach  to  the 
analysis  and  design  of  systems.  Break  down  major 
functions  into  manageable  small,  component  functions. 
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6.  Take  a  disciplined  top-down  approach  to  the 
implementation  of  systems.  Address  the  implementa¬ 
tion  of  major  functions  and  the  resolution  of  major 
potential  problems  before  considering  the  more  pre¬ 
dictable  detailed  systems'  functions. 

7.  Show  the  users  output  that  they  can  understand 
before  final  systems  acceptance.  Break  large,  complex 
systems  into  smaller,  more  manageable  projects  so  that 
users  can  see  results  early  and  can  better  evaluate 
the  progress  and  value  of  systems. 

8.  Evaluate  systems  not  only  in  terms  of  costs  of 
development  and  operation,  but  also  in  terms  of  over¬ 
all  lifetime  costs  and  benefits  [52:32-33]. 

After  laying  a  conceptual  base  for  SA,  Weinberg 
then  identifies  "tools"  used  in  SA.  The  Weinberg  tool  used 
in  our  research  was  the  Data  Flow  Diagram  (DFD) .  "A  graphic 
tool  that  represents  data  flow  and  transforms  in  a  process 
[52:313]."  For  example,  the  DFD  representation  of  updating 
a  master  file  with  individual  transaction  information  would 
look  like: 


Figure  1-5.  Data  Flow  Diagram  Example. 
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AIRCREW  TRAINING  DEVICE  (ATP)  ACQUISITION 

Definitions . 

1.  Aircrew  Training  Device.  AFR  50-11 
defines  an  aircrew  training  device  as  a  synthetic  training 
device  that  is  used  to  support  aircrew  training  programs. 
This  thesis  deals  with  six  of  ten  types  of  ATD  delineated  in 
AFR  50-11:  Part  Task  Trainers  (PTTs) ,  Cockpit  Familiariza¬ 
tion  Trainers  (CFTs) ,  Cockpit  Procedures  Trainers  (CPTs) , 
Mission  Trainers  (MTs) ,  Operational  Flight  Trainers  (OFTs) , 
and  Weapon  System  Trainers  (WSTs)  (46:1). 

2.  System-managed  ATD.  A  device  used  to 
support  a  training  program  for  a  specific,  developing  weapon 
system  is  called  system-managed  (46:3)  and  its  acquisition 
is  the  responsibility  of  the  weapon  system's  program  mana¬ 
ger  (PM)  (46:5) . 

3.  Nonsystem-managed  ATD.  ATD  that  is  pro¬ 
cured  to  support  training  for  either  more  than  one  system 
program  or  for  an  existing  weapon  system,  or  ATD  that  is 
procured  by  an  agency  separate  from  a  weapon  System  Program 
Office  (SPO)  is  considered  a  nonsystem-managed  ATD 

(46:3)  . 

ATD  Acquisition  Process.  There  are  many  ways  that 
needs  for  new  training  devices  may  evolve.  For  instance, 
training  organizations  are  constantly  reviewing  programs  to 
insure  that  their  graduates  are  meeting  Air  Force  needs. 
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These  reviews  are  performed  at  the  task  level  using  the  ISD 
process.  Sometimes  inputs  to  these  reviews  come  in  the  form 
of  individual  supervisors'  reviews  of  graduates  as  well  as 
periodic  meetings  between  training  organizations  and  field 
representatives.  An  output  of  this  process  is  the  identi¬ 
fication  of  training  deficiencies  —  training  skills  that 
the  graduate  should  have  but  does  not.  Further  analysis  by 
the  ISD  process  could  reveal  that  an  optimum  training  media 
mix  should  include  a  new  training  device. 

Essentially,  this  same  process  occurs  during  the 
early  stages  of  a  major  weapon  system  acquisition  in  that 
the  user  ISD  personnel  and  human  factors  engineers  analyze 
the  W/S  and  target  population,  identify  deficiencies,  and 
subsequently  develop  training  programs  with  alternative 
training  media  packages  (43:3).  Other  ways  that  needs  are 
developed  include  obsolescence  of  existing  equipments, 
cost  reduction  opportunities,  and  training  technology 
advances  (26:9). 

For  nonsystem-managed  ATD,  the  need  is  documented 
in  a  Statement  of  Operational  Need  (SON)  (46:3).  This  SON 
should  define  alternative  system  concepts  "in  terms  of 
tasks  or  events  that  can  be  trained  in  each  device  [46:3]," 
and  the  need  should  have  been  developed  using  ISD  proce¬ 
dures  (46:3;  43:3;  48:11).  After  a  Major  Command  (MAJCOM) 
review,  the  SON  is  sent  to  HQ  USAF  for  validation;  and, 
subsequent  to  validation  by  the  appropriate  authority, 
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HQ  USAF  will  issue  a  Program  Management  Directive  (PMD) 
assigning  acquisition  authority  and  additional  program  di¬ 
rection  (47:5).  The  SPO  will  usually  be  ASD's  Deputy  for 
Simulators  (SIMSPO) . 

For  a  system-managed  device,  the  process  is  simi¬ 
lar;  however,  after  the  ATD's  SON  is  validated,  instead  of 
USAF  issuing  a  PMD  for  the  ATD,  the  developing  weapon  sys¬ 
tem's  PMD  and  other  program  documents  will  be  amended  to 
reflect  the  ATD  requirement  and  the  weapon  system  PM  be¬ 
comes  responsible  for  its  acquisition  (46:5).  Then,  either 
the  human  factors  engineers  perform  additional  task  analy¬ 
sis  using  ISD  procedures  or  the  task  analysis  effort  is 
contracted  (4;  43:3).  Alternative  ATD  system  designs  begin 
to  evolve  that  weigh  user  prioritized  training  objectives 
against  cost  (4) . 

At  this  point,  the  system-managed  and  nonsystem- 
managed  ATD  are  both  undergoing  similar  activities  based 
on  and  justified  by  ISD  developed  training  objectives.  The 
nonsystem-managed  equipment  will  progress  through  the  nor¬ 
mal  800-series  regulations'  acquisition  process  under  SIM¬ 
SPO  direction.  The  system-managed  device  will  undergo  a 
similar  process;  however,  after  formal  direction  to  proceed 
into  the  demonstration  and  validation  phase,  the  PM  may 
elect  to  manage  the  development  or  to  transfer  the  manage¬ 
ment  of  the  ATD  development  effort  to  the  SIMSPO  (1:2). 

If  the  program  is  transferred  to  the  SIMSPO,  the 
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weapon  system  PM  and  SIMSPO  will  document  their  respective 
organizational  roles,  responsibilities,  and  procedures  in 
a  Memorandum  of  Agreement  (MOA)  (1:2).  Essentially,  the 
SIMSPO  assumes  all  the  management  aspects  associated  with 
the  ATD  and  will  tailor  its  acquisition  program  to  coin¬ 
cide  with  the  major  weapon  system's  since  the  ATD  must 
develop  dependent  on  such  data  as  the  evolving  performance 
and  physical  characteristics  of  the  major  weapon  system  as 
well  as  its  maintenance  and  operational  concepts.  In  the 
MOA,  the  PM  will,  therefore,  define  the  transfer  of  data 
from  the  major  weapon  system  contractors,  users,  and  SPO, 
to  the  ATD's  contractors  and  SIMSPO  (1:2). 

ATD  acquisition  programs  follow  the  typical  acqui¬ 
sition  process.  The  weapon  system  acquisition  process  lay¬ 
ers  over  the  ATD  process  although  the  weapon  system's  acqui¬ 
sition  is  initiated  before  and  ends  after  the  ATD's.  Figure 
1-6  depicts  the  time-line  for  a  typical  ATD  development  by 
SIMSPO,  and  Figure  1-7  depicts  the  major  data  inputs/out¬ 
puts  and  processes  involved  in  a  typical  system-managed  ATD 
acquisition  program. 

The  constraints  on  the  current  training  system  dev¬ 
elopment  and  ATD  acquisition  process  are: 

1.  ISD  task  analysis  is  not  compatible  with  the 
current  ATD  acquisition  process. 

2.  The  current  process  is  not  designed  to  encourage 
iterations  of  the  training  analysis  to  improve  the  training 
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Figure  1-6.  ATD  Time-line  (10). 


system  and  ATD  definition. 

3.  The  current  system  does  not  prevent  "goldplat- 
ing"  of  ATDs  with  unnecessary  capabilities. 

4 .  The  current  system  does  not  encourage  the  deci¬ 
sion  maker  to  search  for  optimum  training  system  and  ATD 
configurations . 

5.  The  current  system  does  not  deliver  ATDs  to  the 
user  at  IOC. 

6.  The  ISD  process  is  not  amenable  to  the  time  con¬ 
straints  imposed  by  the  weapon  system  schedule.  . 

RESEARCH  QUESTION 

What  type  of  structured  programming  model  can  be 
developed  that  will  both  enable  accurate  assessments  of  ATD 
requirements  early  in  the  acquisition  of  a  major  weapon 
system  and  be  useful  for  communicating  these  ATD  require¬ 
ments  to  the  acquisition  specification  writer  and  decision 
maker? 

SCOPE 

This  thesis  will  address  the  acquisition  of  Aircrew 
Training  Devices  (ATDs) ,  as  defined  in  this  chapter.  The 
acquisition  of  training  devices  for  maintenance  has  some 
differences  not  found  with  ATD  and  will  not  be  analyzed 
as  part  of  this  thesis.  Further,  we  will  study  the  acqui¬ 
sition  cycle  for  ATD  from  the  requirements  determination 
process  through  development  of  a  product  specification. 
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CHAPTER  2 


METHODOLOGY 

We  will  analyze  the  current  training  system  and  ATD 
development  process,  and  develop  a  proposed  system  model 
via  our  modification  of  Victor  Weinberg's  structured  meth¬ 
odology  (52:Ch.4).  The  modifications  enable  us  to  use 
this  processing  design  methodology  for  this  research.  Our 
methodology  consists  of  eight  phases: 

1.  Problem  definition. 

2.  Description  of  current  system, 

3.  Criterion  definition, 

4.  Preliminary  approaches  to  problem, 

5.  Solution  overview, 

6.  Detailed  design, 

7.  Validation  walkthrough, 

8.  Post-evaluation. 

This  chapter's  two  sections  detail  each  phase  in 
the  methodology  and  then  explain  how  we  will  present  them 
in  this  research  project. 

SECTION  1:  CONCEPTUAL  FRAMEWORK 

In  this  section,  we  define  each  of  our  methodology's 
eight  phases  by  reporting  the  following  for  each  phase: 
the  phase's  purpose,  major  parts,  and  inputs/outputs 
(if  identified) . 
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PHASE  ONE — PROBLEM  DEFINITION.  Weinberg  identified 
phase  one's  object  and  key  output  as  "to  identify  an  exist¬ 
ing  or  anticipated  problem  [52:291]."  In  this  phase  the 
following  subjects  are  discussed  to  identify  a  problem  and 
to  lay  an  information  base  for  the  other  seven  phases: 

1.  Background  data  on  related  topics, 

2.  Indications  of  a  problem, 

3.  Problem  statement,  , 

4.  Constraints  associated  with  the  problem. 

Inputs  for  this  section  come  from  several  sources. 

The  primary  source  of  background  information  and  constraints 
is  Air  Force  Regulations  and  publications.  Information 
about  indications  of  a  problem  comes  from  interviews, 
studies,  meeting  minutes,  and  regulations.  Once  we  under¬ 
stand  background  data  and  the  indications  of  the  problem, 
we  focus  our  attention  on  describing  the  current  system. 

PHASE  TWO — DESCRIPTION  OF  CURRENT  SYSTEM.  The 
purpose  of  phase  two  is  to  show  how  the  process  should 
and  does  operate.  It  answers  the  question  of  how  the  Air 
Force  identifies  and  acquires  ATDs.  A  review  of  ideal 
and  actual  (from  phase  one)  processes  will  identify  limi¬ 
tations  which  prevent  the  acquisition  process  from  working 
as  it  should. 

The  following  areas  will  be  reviewed  to  describe 
the  current  ATD  process : 
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1 .  Description  of  ATD  acquisition  process , 

2.  Ideal  flow  of  the  process  information, 

3.  Comparison  with  indications  of  problem  (from 
phase  one) , 

4.  Statement  of  current  system  limitations. 

The  limitations  on  the  current  system  provide  the  key  input 
to  the  next  phase. 

PHASE  THREE — CRITERION  DEFINITION.  The  purpose  of 
phase  three  is  to  identify  the  "yardstick" — the  criteria — 
against  which  to  judge  alternative,  preliminary  approaches 
to  the  problem  solution  and  our  proposed  solution.  Our 
definition  process  starts  with  the  premise  that  ATDs  must 
be  acquired  which  use  design  inputs  from  an  ISD-type  process 
very  similar  to  and  compatible  with  Air  Force  ISD.  Our 
next  premise  is  that  the  airframe  acquisition  effort,  be¬ 
cause  of  its  relative  size  and  resources,  is  not  subject  to 
major  changes  to  accommodate  the  training  and  ATD  efforts — 
the  tail  cannot  wag  the  dog.  Next,  we  assume  that  if  the 
identified  limitations  can  be  eliminated,  a  new  system  would 
equate  to  an  ideal  training  and  acquisition  process.  Since 
a  perfect  solution  is  improbable,  our  goal  is  to  ameliorate 
the  identified  limitations.  Therefore,  as  Weinberg  sug¬ 
gests,  we  develop  criteria  that  correspond  to  identified 
limitations. 

Once  we  establish  the  criteria,  we  validate  and 
revise  them  by  using  the  following  procedure.  We  select 
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a  small  group  of  experts  available  at  Wright-Patterson 
AFB  OH  which  represent  the  following  areas: 

1.  ISD  analysis/research, 

2.  ATD  acquisition  management, 

3.  ATD  acquisition  engineering. 

In  structured  interviews  we  first  explain  our  research 
topic  and  problem  statement.  Next,  we  present  them  with 
the  criteria  and  ask  for  their  comments  in  the  following 
areas : 

1.  Are  the  criteria  clear  and  understandable? 

2.  Are  any  major  problem  areas/limitations 
missing  from  the  criteria? 

3.  Are  there  other  comments? 

After  the  interviews,  we  revise  the  criteria  to  reflect 
these  experts '  comments . 

PHASE  FOUR — PRELIMINARY  APPROACHES  TO  PROBLEM.  The 
purpose  of  phase  four  is  to  determine  if  solutions  or  ideas 
which  meet  our  criteria  exist  in  current  sources.  In  this 
phase  we: 

1.  Identify  sources  familiar  with  the  problem  and/ 
or  potential  solutions  , 

2.  Report  a  review  of  sources, 

3.  Compare  methods  reviewed  against  criteria, 

4.  Determine  if  solutions  are  possible  using  only 
existing  methods. 

The  output  of  this  phase  would  serve  one  of  two 
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functions.  First,  if  a  reviewed  method  satisfies  all  cri¬ 
teria  it  could  conceivably  solve  the  problem  statement. 

The  second  and  more  probable  event  is  that  no  method 
satisfies  all  criteria,  and  the  method  serves  as  an  input 
to  a  system  model  developed  by  the  authors. 

PHASE  FIVE — SOLUTION  OVERVIEW.  The  purpose  of 
phase  five  is  to  begin  the  system  model  development  and 
reporting  phase.  Output  is  the  features  of  the  proposed 
solution  developed  by  the  authors.  While  the  reader  must 
wait  until  phase  six  to  see  the  detailed  step-by-step 
model,  phase  five  presents  and  explains  the  features  that 
the  system  contains  and  how  the  system  model  satisfies  the 
criteria. 

There  are  four  inputs  to  phase  five.  The  first 
input  is  the  criteria  developed  in  phase  three.  The  second 
input  is  the  methods  reviewed  in  phase  four.  The  third 
input  is  phase  four's  comparison  of  criteria  with  reviewed 
methods.  And  the  fourth  input  is  the  knowledge  and  ideas 
of  the  authors. 

Using  the  inputs,  the  authors  develop  their  pro¬ 
posed  system  model  via  the  following  steps.  First,  the 
phase  four  methods  of  no  use  (helped  solve  no  limitation/ 
criterion)  are  eliminated.  Second,  a  set  of  method  fam¬ 
ilies  are  established  which  combine  like  methods  from 
various  articles  and  sources.  Third,  method  families  are 
identified  with  training  system  and  ATD  development  phases, 
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if  possible.  Fourth,  mutually  exclusive  and  complementary 
method  families  are  identified.  The  model  building  then 
enters  phase  six — detailed  design.  In  the  thesis'  text  we 
place  the  comparison  of  criteria  and  proposed  solution 
after  phase  six. 

PHASE  SIX — DETAILED  DESIGN.  The  purpose  of  phase 
six  is  to  complete  the  model  development  and  reporting 
sequence  begun  in  phase  five.  The  unique  aspect  of  phase 
six  is  that  the  features  are  represented  in  DFD  form.  The 
inputs  come  from  phase  five,  and  the  output  is  a  three- 
level  DFD.  The  first  (summary)  level  DFD  shows  the  major 
processes  and  their  inputs /outputs .  The  next  two  levels 
show  each  process  in  increasing  detail.  As  suggested  by 
Weinberg,  we  stop  the  DFD  process  when  identifiable 
functions  which  could  be  easily  assigned  to  organization 
units  are  reached.  Following  the  DFDs  and  explanation,  the 
model  is  compared  with  the  criteria. 

PHASE  SEVEN-VALIDATION  WALKTHROUGH.  The  purpose 
of  phase  seven  is  to  subject  our  system  model  to  the 
training  and  ATD  development  community.  We  validate  our 
system  model  with  a  sample  of  experts  from  four  functional 
areas: 

1.  ISD  development/research, 

2.  MAJCOM  liaison, 

3.  ATD  development  management, 

4.  ATD  development  engineering. 
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We  are  able  to  limit  our  validation  to  Wright-Patterson 
AFB  OH  because  it  is  the  center  of  aircraft  and  ATD  acqui¬ 
sition,  as  indicated  by  the  fact  that  over  50%  of  AFSC's 
annual  budget  is  spent  by  Wright-Patterson' s  Aeronautical 
Systems  Division.  ASD  is  not  only  the  Air  Force's  manager 
of  aircraft  research  and  development;  but  the  SIMSPO,  an 
ASD  directorate,  is  charged  with  Air  Force  ATD  develop¬ 
ment  and  acquisition.  In  addition,  MAJCOM  liaison  offices 
located  at  ASD  provide  the  user  prospective.  Because  of 
the  concentration  of  acquisition  activity  at  Wright-Patter¬ 
son  AFB,  it  offers  an  excellent  location  for  the  valida¬ 
tion  walkthrough.  We  insure  a  degree  of  interviewee  exper¬ 
tise  by  defining  an  expert  as  a  person  who  meets  at  least 
one  of  the  following  standards: 

1.  A  published  author  in  the  fields  of  training 
or  ATD  development, 

2.  An  incumbent  in  a  training  or  ATD  development 
job  for  at  least  one  year, 

3.  A  masters  degree  or  higher  in  fields  related 
to  training  or  ATD  development . 

We  decided  that  a  structured  interview  best  met 
our  needs  and  developed  the  following  procedure.  After 
introducing  ourselves  and  our  research  topic,  we  ask  the 
interviewee  to  rate  the  current  Air  Force  training  and  ATD 
development  system  using  a  set  of  prepared  statements  and 
Likert  five-point  scales.  Next,  we  review  our  proposed 
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system  (draft  of  Chapter  Four)  with  the  interviewee  and 
ask  him  to  rate  the  system  using  the  same  set  of  statements 
and  Likert  scales.  Finally,  we  ask  the  interviewee  for 
general  comments  about  the  system  and  criteria. 

The  statements  are  written  and  validated  in  the 
following  manner.  At  least  one  statement  per  criterion  is 
composed  with  a  Likert  scale  following  each  question 
(39:325-326).  The  statements  are  validated  by  the  expert 
opinion  of  members  of  the  Air  Force  Institute  of  Technology 
(AFIT)  faculty. 

The  prime  objective  of  our  validation  walkthrough 
is  to  record  opinions  of  experts  representative  of  the  Air 
Force  training  and  ATD  communities.  We  will  ask  Mr.  Robert 
E.  Coward,  Special  Assistant,  Deputy  for  Simulators,  to 
recommend  a  list  of  experts  that,  in  his  opinion,  meet 
our  standards.  We  feel  that  Mr.  Coward  is  objective  in  his 
recommendations  because  (1)  his  unique  position  as  Special 
Assistant,  he  is  outside  the  management  chain;  (2)  he  is 
organizationally  independent  of  the  other  areas  in  which 
we  will  seek  recommendations.  We  select  Mr.  Coward  as 
our  expert  in  the  area  of  ATD  management  based  on  his  cre¬ 
dentials,  and  only  ask  him  for  experts  in  the  other  three 
areas,  and  because  of  his  extensive  experience  in  ATD 
acquisition  and  participation  in  training  and  ATD  seminars, 
Mr.  Coward  is  acquainted  with  a  large  portion  of  the  experts 
in  the  field. 
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After  the  interviews,  we  test  the  following  hypoth¬ 


esis  for  each  statement/criterion: 

Hq:  The  training  and  ATD  development  system  pro- 


proposed  by  the  authors  shows  no  improvement  over  the  current 
Air  Force  training  and  ATD  development  system. 

H  :  The  training  and  ATD  development  system  pro- 

cl 

posed  by  the  authors  is  a  significant  improvement  over  the 
current  training  and  ATD  development  system.  Stated  math¬ 
ematically: 


H  :  U  -  U  <  0 
o  x  y  — 


H  :  U  -  U  >0 
a  x  y 


where:  =  mean  score  for  the  proposed  system 


Uy  =  mean  score  for  the  current  Air  Force  system 


(28;  53:450-452). 


The  hypothesis  is  tested  with  a  Paired  Sample 
T-Test.  The  paired  scores  for  the  interviewees  are  used 
to  develop  a  t-statistic  using  the  following  equation: 
t  =  [5  (n*5)  ]  /sQ 

where:  6  ■  ?  (X^  -  Y^)/n 

*  the  mean  of  the  differences  between  the  paired 

scores . 

X.  *  i  th  score  for  the  proposed  system 

Yi  =  i  th  score  for  the  current  Air  Force  system 
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n  =  the  sample  size 

sD  =  the  standard  deviation  of  the  difference  be¬ 
tween  the  paired  values  using  the  n-1  method. 

The  Texas  Instruments,  Applied  Statistics  Solid  State 
Software  module,  program  number  13  (t-Statistic  Evaluation) 
is  used  for  the  t-statistic  computation  (38:4-12  to  4-15). 
The  null  hypothesis  rejection  region  is  set  at  the  90% 
confidence  level  with  n-1  degrees  of  freedom  (53:450-452). 

PHASE  EIGHT — POST-EVALUATION .  The  goal  of  phase 

\  --  -  - 

eight  is  to  successfully  conclude  the  research  effort. 

The  chapter's  objectives  are  to  draw  final  conclusions 
about  our  system  model  and  to  make  recommendations.  Our 
first  objective  is  to  make  final  conclusions  and  recom¬ 
mendations.  Our  second  objective  is  to  discuss,  in  general 
terms,  implementation  based  on  our  perceptions  of  the 
current  Air  Force  and  training  environments.  Our  third 
goal  is  to  identify  areas  needing  continued  research. 

SECTION  2:  INTEGRATION  OF  PHASES  INTO  CHAPTERS 

Our  research  project  is  our  exercise  of  the  eight- 
phase  methodology.  The  chapter  and  phase  arrangement,  and 
rational  are  discussed.  There  are  six  chapters: 

1.  Chapter  One— Introduction.  We  include  phases 
one  (Problem  Definition)  and  two  (Description  of  Current 
System)  in  this  chapter.  Our  objectives  for  the  chapter  i:- 
for  the  reader  to  understand: 
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A.  That  a  problem  exists, 

B.  The  current  environment  (or  background) 


which  includes  ISD,  W/S,  and  ATD  acquisition, 

C.  That  Structured  Analysis  is  an  orderly 
method  to  solve  complex  problems, 

D.  That  the  current,  actual  ISD/ATD  develop¬ 
ment  faces  limitations  which  cause  problem  indications. 

2.  Chapter  Two — Methodology.  No  phases  are  in 
this  chapter,  but  the  chapter  is  vital  to  the  research. 

Our  objective  for  the  chapter  is  for  the  reader  to  under¬ 
stand: 

A.  Each  phase  of  the  structured  methodology, 

B.  How  the  chapters  and  phases  are  arranged. 

3.  Chapter  Three— Initial  Research  and  Findings. 
We  include  phases  three  (Criterion  Definition)  and  four 
(Preliminary  Approaches  to  Problem)  in  this  chapter.  Our 
objective  for  the  chapter  is  for  the  reader  to  understand: 

A.  The  criteria — how  they  are  developed,  what 
they  are,  and  how  they  are  used/graded j 

B.  The  current  knowledge  in  the  area  --its 
sources,  contents,  and  comparison  with  the  criterion. 

4.  Chapter  Four — Model  Builaxng.  We  include 
phases  five  (Solution  Overview)  and  six  (Detailed  Design) 
in  this  chapter. '  Our  objective  is  to  present  our  model 
and  compare  it  with  the  criteria. 

5.  Chapter  Five- -Validation  Walkthrough.  We 
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include  phase  seven  (Validation  Walkthrough)  in  this  chap¬ 
ter.  Our  objective  is  to  determine  the  validity  of  our 
model  based  on  expert  opinion. 

6.  Chapter  Six — Conclusions  and  Recommendations. 

We  include  phase  eight  (Post-Evaluation)  in  this  chapter. 

Our  objectives  are  to  draw  final  conclusions  and  to 
recommend  an  implementation  plan  and  further  research. 

SECTION  3:  RESEARCH  OUTPUTS 

In  this  section,  we  summarize  the  research  products. 
There  are  three  outputs  of  this  research: 

SYSTEM  MODEL.  We  present  a  proposed  system  model 
for  training  and  ATD  development  for  emerging  W/Ss.  First, 
we  present  the  features  which  are  imbedded  in  the  system 
model.  Second,  we  present  the  logical  processes  needed  to 
develop  training  and  ATD  systems. 

VALIDATION  WALKTHROUGH.  We  then  present  the  valida¬ 
tion  results  for  our  system  model.  The  result  will  indi¬ 
cate  if  training  and  ATD  development  experts  feel  our  system 
model  is  or  is  not  an  improvement  over  the  current  Air  Force 
system. 

CONCLUSIONS  AND  RECOMMENDATIONS.  We  then  present 
our  final  conclusions  on  the  training  and  ATD  development 
system,  make  general  implementation  recommendations,  and 
suggest  areas  needing  additional  research. 
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CHAPTER  3 


INITIAL  FINDINGS  AND  CONCLUSIONS 

Having  completed  phases  one  and  two  in  Chapter  1  and 
having  developed  our  methodology  in  Chapter  2 ,  we  continue 
through  the  structured  methodology  in  Chapter  3  by,  first, 
defining  our  research  criterion  —  phase  three.  Second,  we 
report  on  preliminary  approaches  to  problem  solution  — 
phase  four.  Third,  we  make  some  initial  conclusions  about  the 
preliminary  approaches. 

SECTION  1:  PHASE  THREE— CRITERION  DEFINITION 

As  described  in  Chapter  2 ,  Weinberg  recommends  that 
criteria  be  developed  using  the  following  sequential  pro¬ 
cess: 

1.  Identify  and  gain  an  understanding  of  the 
problems  with  the  current  system. 

2.  Transform  the  problems  into  system  limitations. 
This  is  not  a  one-to-one  process  since  a  system  limitation 
may  represent  more  than  one  problem  and  vice  versa . 

3.  Finally,  because  these  limitations  are  causing 
systemic  problems,  the  prime  objective  of  the  system 
design  effort  is  to  eliminate  these  limitations  (see  Re¬ 
search  Objectives  in  Chapter  1).  Therefore,  define  criteria 
that,  when  met,  eliminate  or  ameliorate  the  limitations. 

The  same  is  true  about  criteria  and  limitations  that  is 
true  about  limitations  and  problems:  the  transformation 
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from  limitation  to  criterion  is  not  one-to-one  but  repre¬ 
sents  a  mental  process  that  enhances  the  conceptual  under¬ 
standing  of  the  system. 

The  following  are  the  six  criteria  based  on  the 
problems  and  limitations  identified  in  Chapter  1.  We  will 
use  these  criteria  to  evaluate  both  the  potential  solutions 
in  Section  2  of  this  chapter  and  our  own  system  model  in 
Chapter  4 : 

1.  Criterion:  The  system  output  must  produce  ATD 
requirements  for  an  emerging  weapon  system  based  on  the 
information  available  at  the  time  the  analysis  is  required. 
Explanation:  This  means  that  the  system  must  identify  ATD 
functional  needs  based  on  the  data  available  in  the  concep¬ 
tual  and  validation  phases  and  early  in  the  FSD  phase  of  the 
W/S  development  program.  If  uncertainties  exist  with  the 
W/S  data,  then  the  ATD  need  must  address  these  uncertain¬ 
ties.  The  main  point  here  is  that  the  ATD  need  identifi¬ 
cation  cannot  and  should  not  be  delayed  until  the  FSD 
phase  of  the  W/S  is  completed  and  the  W/S  data  is  firm. 

2.  Criterion:  After  the  initial  iteration  of  ATD 

requirements,  the  system  must  revise /improve  the  ATD  re¬ 
quirements  as  new  information  becomes  available.  Explana¬ 
tion:  If  an  ATD  is  functionally  defined  prior  to  the  W/S 

data  being  frozen,  then  we  run  the  risk  of  delivering  an 
ATD  that  does  not  replicate  the  fielded  W/S.  Therefore,  as 
the  data  changes,  the  ATD  definition  must  change  in  those 


areas  that  would  limit  the  training  effectiveness  of  the 
device . 

3.  Criterion:  The  system  must  produce  ATDs  with 
only  those  capabilities  essential  to  training  identified 
tasks  (i.e.,  prevent  "goldplating").  Explanation:  We 
define  "goldplating"  as  knowingly  designing  an  ATD  which 
has  excess  capabilities  and  is  excessively  expensive.  It 
is  a  hedge  against  the  uncertainty  of  ill-defined  require¬ 
ments  and  must  be  eliminated. 

4.  Criterion:  The  system  must  aid  ATD  acquisition 
decision  makers  (for  example:  user,  W/S  SPO,  and  SIMSPO) 
with  cost/capability /tasks-trainable  tradeoffs.  Explana¬ 
tion:  The  current  system  does  not  encourage  the  search  for 
the  optimum  mixture  of  ATD  cost/capability /tasks-trainable . 

A  system  must  encourage  the  decision  makers  to  evaluate 
numerous  ATD  mixtures  and  to  select  the  optimum  ATD. 

5.  Criterion:  The  system  must  improve  the  prob¬ 
ability  of  ATD  delivery  at  IOC  or  earlier  than  the  current 
system.  Explanation:  Assuming  that  to  meet  the  W/S  IOC 
with  training  devices  is  desirable  and  recognizing  the  fact 
that  it  is  mandated  by  regulation,  the  training  ATD  acquisi¬ 
tion  system  must  enable  the  delivery  of  training  effective 
ATDs  prior  to  the  IOC  of  the  W/S. 

6.  Criterion:  The  system  must  shorten  the  time 
between  the  start  of  W/S  analysis  and  completion  of  final 
ATD  requirements.  Explanation:  Training  experts  claim  that 
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the  time  to  complete  the  ISD  process  is  relatively  fixed, 
but  if  the  ISD  process  can  be  streamlined,  it  will  better 
fit  the  current  W/S  acquisition  process. 

SECTION  2:  PHASE  FOUR— PRELIMINARY  APPROACHES  TO  PROBLEM 

The  following  are  potential  sources  of  data  and 
serve  as  major  subsections: 

1.  Military  services'  ATD  communities, 

2.  Military  study  offices/labs, 

3.  Reports /studies  contracted  by  the  services, 

4.  Other  articles/papers. 

MILITARY  SERVICES'  ATD  COMMUNITIES.  Both  the  Army 
and  Navy  have  ATD  acquisition  functions.  TDY  visits  to  the 
Naval  Training  Equipment  Center  (NTEC)  and  the  Army  Aviation 
Center  were  undertaken  and  are  reported  here.  While  Naval 
ATD  acquisition  is  centralized  at  NTEC  and  one  TDY  provides 
a  complete  view  of  their  process.  Army  ATD  acquisition  is 
decentralized  (both  in  location  and  command  function)  and 
one  TDY  plus  three  articles  are  necessary  to  give  a  complete 
view  of  the  Army  process. 

Army  Aviation  Center.  The  current  Army  ATD  develop¬ 
ment  program  is  smaller  and  less  developed  than  the  Air 
Force's  ATD  development  program.  This  is  not  surprising, 
nor  necessarily  bad,  since  aviation  is  only  one  of  many 
functions  assigned  to  the  Army  (examples  of  other  functions 
being  infantry,  armor,  airborne,  etc.).  In  addition,  while 
Air  Force  aircraft  are  "center  stage"  in  our  operations, 
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Army  aviation  units  are  subservient  to  their  proponent 
commands.  For  example,  the  UH-60A  helicopter's  mission  is 
to  support  the  infantry. 

The  current  Army  "Life  Cycle  System  Management 
Model"  does  include  training  and  ATD  development  as  part  of 
V/S  development,  but  actual  practice  faces  three  problem 
areas.  First,  there  is  no  unifying  force  behind  ATD  devel¬ 
opment.  That  responsibility  is  spread  over  three  commands 
(Training  and  Doctrine,  Army  Material  Development  and  Read¬ 
iness,  and  the  field  commands)  and  various  agencies 
(34:7;  14).  Second,  in  some  cases  a  systematic  training 
development  effort  is  not  adequate  due  to  time/money /man¬ 
agement  constraints  (34:8;  14;  18:3-4)  .  Third,  the  training 
subsystem  is  lumped  in  with  other  logistics  activities  while 
some  interviewees  felt  that  training  should  be  separate 
(14)  . 

A  brief  review  of  the  Army  ATD  procurement  system 
reveals  that: 

1.  ISD  principles  are  directed  to  be  used, 

2.  Desired  and  actual  activities  are  not  possible 
because  of  time /money /management  constraints, 

3.  There  is  a  three  to  five  year  gap  between  IOC 
and  simulator  delivery  (for  example,  the  UH-60A's  IOC  was 
May  1979  and  ATD  operational  tests  are  not  scheduled  for 
completion  until  mid-1982)  (16;  27:496-499;  34:7-9), 

4.  Additional  study  is  needed/desired  (18:4). 
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Naval  Training  Equipment  Center  (NTEC) .  The  Navy 
training  device  acquisition  process,  like  the  Air  Force's, 
begins  with  a  training  deficiency,  training  problem,  or 
need  for  devices  to  support  an  emerging  weapon  system 
(51:2).  NTEC  then  performs  a  Training  Situation  Analysis 
(TSA) .  The  TSA  can  take  three  forms  (17) : 

1.  ISD  analysis:  This  formal  process  is  used  to 
identify  training  media  when  no  predetermination  has  been 
made  what  media  are  actually  needed  to  support  the  instruc¬ 
tional  program. 

2.  Training  Situation  Analysis  with  Prerequisites: 
This  formal  process  is  used  to  identify  the  training  media 
when  the  segment  of  training  to  be  addressed  is  firmly 
identified.  It  differs  from  an  ISD  analysis  because  this 
approach  does  not  require  a  task  analysis. 

3.  Front  End  Analysis  (FEA) :  This  is  not  a  formal 
approach  but  requires  "inventiveness  and  initiative  by  the 
analyst  [50:2]."  The  user  has  either  identified  the  need 
for  a  specific  type  of  medium  or  has  identified  a  specific 
segment  of  training  that  needs  to  be  addressed.  The  three 
processes  recognize  that  different  situations  require 
varying  degrees  of  analysis  and  expenditure  of  resources 
and  time. 

The  output  of  a  TSA  is  a  functional  description  of 
the  device/media  needed.  The  functional  description  is  then 
refined  by  the  project  team  (training  specialist,  design 
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engineer,  Integrated  Logistics  Support  manager) 

through  further  analysis.  Four  documents  conclude  the  steps 

of  this  process  (50) : 


1.  Functional  Statement  (FS) :  This  is  a  brief, 
concise  statement  of  a  device  requirement  that  documents 
the  desired  behavioral  objectives  and  device  physical  and 
functional  characteristics.  This  document  receives  user/ 
NTEC  review. 

2.  Functional  Description  (FD) :  This  document 
builds  on  the  FS  by  providing  additional  specificity  and 
performance  parameters.  This  document  must  receive  user/ 
NTEC  approval  before  proceeding. 

3.  Mini-Military  Characteristics  (Mini-MC) :  This 
document  builds  on  the  FD  by  providing  for  more  detailed 
functional  requirements  and  addressing  ILS.  The  Mini-MC 
becomes  the  source  document  for  the  Mini-design  effort. 
CJser/NTEC  approval  concludes  this  step. 

4.  Detail  Military  Characteristics  (Detail  MC) : 
This  document  adds  additional  detail  to  the  device  func¬ 
tional  and  physical  specifications.  It  becomes  the  source 
document  for  the  design  of  the  device  and  receives  user/ 

ISD  review. 

The  Navy  process  has  two  noteworthy  strengths. 
First,  the  review  of  each  document  by  users  and  ISD  per¬ 
sonnel  is  conducted  to  reduce  the  probability  of  a  disjoint 
between  the  item  that  is  finally  delivered  and  the  device 
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that  the  training  program  needs.  Second,  this  review  is 
facilitated  by  the  collocation  of  ISD  personnel  and  project 
engineers  on  the  same  base. 

MILITARY  STUDY  OFFICES /LABS .  This  subsection  re¬ 
views  the  military  supported  study  offices  and  laboratories. 
The  services'  study  offices/ labs  are: 

1.  U.S.  Air  Force  Human  Resources  Laboratory 
(AFHRL) ,  Brooks  AFB  TX. 

2.  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  (ARI)  Field  Unit,  Fort  Rucker  AL. 

We  limit  our  review  in  this  subsection  to  papers  written  by 
office/laboratory  members. 

Cream,  Eggemeier,  and  Klein.  The  Cream,  et  al , 
article  discusses  several  issues  related  to  ATD  design.  The 
issues  of  most  interest  to  us  were  defining  training  re¬ 
quirements  and  determining  ATD  capabilities  based  on  train¬ 
ing  requirements. 

This  article  proposes  "an  integrated  system  develop¬ 
ment  process  involving  users,  training  psychologists,  and 
simulation  engineers  [6:146]."  This  is  accomplished  by  the 
coordinated  efforts  of  the  three  groups  with  the  training 
psychologist  coordinating  group  activities.  The  group's 
activities  are: 

1.  Initiate  task  analysis  with  a  complete  stimu¬ 
lus — response  (S/R)  listing  for  each  control  and  display. 
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2.  Determine  ATD  capabilities  to  support  the  trained 

tasks. 

3.  Rate  each  task  by  three  dimensions:  criticality, 
frequency,  and  difficulty  of  performance  (C/F/D) .  Then  de¬ 
lete  uniformly  low  ranked  tasks  and  keep  uniformly  high 
ranked  tasks. 

4.  Review  remaining  tasks  which  received  mixed 
C/F/D  ratings.  The  relative  importance  of  each  factor  is 
weighted. 

The  authors  briefly  mention  emerging  weapon  systems. 
They  identify  lack  of  weapon  system  knowledge  as  a  problem 
and  suggest  basing  data  on  existing  equipment  and  tasks. 
However,  they  do  not  present  a  specific  methodology  for  the 
emerging  W/S  (6:146-148). 

Finally,  the  authors  discuss  the  impact  that  training 
requirements  have  on  ATD  capabilities  including:  fidelity 
decisions,  performance  measures,  instructional  capability, 
and  crew  coordination. 

Army  Research  Institute.  The  ARI  Field  Unit,  Fort 
Rucker  AL,  provides  the  Army  Aviation  Center  with  studies 
and  research  related  to  aviation  training.  When  asked  to 
study  training  requirements  for  new  helicopters,  ARI 
awarded/supervised  a  contract  to  analyze  operator's  task 
and  training  requirements.  The  contract  Statement  of  Work 
(SOW)  for  the  mission  analysis  identifies  a  five  step  pro¬ 
cess  to  develop  training  requirements  based  upon  the  pro- 
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jected  use  of  the  aircraft  and  preliminary  equipment  lists. 
The  ARI  process  is  to: 


1.  Develop  mission  profiles  showing  how  each  major 
equipment  system  is  employed  throughout  the  missions  and 
mission  segments. 

2.  Identify  and  characterize,  for  each  mission 
segment,  equipment  use  in  terms  of  operator-equipment  func¬ 
tional  interactions. 

3.  Allocate  equipment  operation  functions  between 
and  among  crewmembers  to  achieve  optimum  workload  and  per¬ 
formance  outcomes. 

4.  Conduct  detailed  analyses  of  all  tasks  and  task 
performance  elements  for  each  crewmember  in  each  system. 

5.  Analyze  training  requirements  for  each  crew¬ 
member  in  each  system. 

The  end  result  of  the  analysis  is  a  set  of  training  objec¬ 
tives  based  on  information  available  early  in  the  weapon 
system  acquisition.  Essentially,  this  procedure  is  ISD 
with  a  synthetic  task  analysis.  The  final  report  of  this 
project  is  due  in  January  1983  (41).  The  basis  of  this 
type  of  process  is  also  found  in  a  NATO  study,  discussed 
later. 

REPORTS /STUDIES  CONTRACTED  BY  THE  SERVICES.  Numer¬ 
ous  studies  have  been  contracted  to  private  firms  by  the 
services  to  study/report  on  aspects  of  ATD  acquisition  and 
training.  We  review  those  reports  which  are  germane  to  our 
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research. 


Wallace  W.  Prophet.  In  1966,  Wallace  W.  Prophet, 
Director  of  Research  of  George  Washington  University's 
Human  Resources  Research  Office  (HumRRO) ,  Division  No.  6 
(Aviation)  presented  a  paper  on  the  importance  of  training 
requirements  in  the  design  of  ATDs.  We  note  that  HumRRO 
was  a  major  source  of  information  for  Army  aviation  training 
in  the  1960s  and  1970s.  Mr.  Prophet's  thesis  is  that  the 
full  potential  of  ATDs  is  not  being  realized  because  of  poor 
design  procedures  (31  :1). 

The  author  contends  that  better  ATDs  are  possible 
if: 


1.  Training  devices  should  be  built  only  after  care¬ 
ful  analysis  of  the  training  requirements. 

2.  Analysis  of  training  requirements  should  be  pri¬ 
marily  psychological,  rather  than  engineering  in  nature. 

3.  Determination  of  device  characteristics  should 
be  oriented  toward  providing  task  fidelity,  with  phy¬ 
sical  fidelity  being  provided  only  as  necessary  to  task 
fidelity. 

4.  Synthetic  training  programs  snould  receive  as 
careful  consideration  in  their  development  as  do  the 
trainers  themselves. 

5.  Devices  and  synthetic  training  programs  should 
be  tested  for  training  effectiveness  just  as  aircraft 
are  tested. 

6.  Training  devices  and  simulators  can  provide  an 
effective  means  of  reinforcing  infrequently  occurring 
behaviors  [31:8]. 

We  include  Mr.  Prophet's  paper  because: 

1.  it  confirms  that  the  ATD  problem  has  been  known 
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for  many  years. 

2.  the  answers  to  the  ATD  acquisition  problem  pro¬ 
posed  by  Prophet  are  echoed  in  recent  studies . 

3.  it  appears  that  implementation  of  solutions  is 
the  most  difficult  aspect  of  the  problem. 

STREPS  Report.  The  Simulator  Training  Requirements 
and  Engineering  Design  Study  (STREDS)  written  by  Technology 
Incorporated  is  an  example  of  current  research  in  training 
and  ATD  acquisition. 

The  purpose  of  this  study  was  to  develop  improved 
methods  for  Front  End  Analysis  of  training  requirements 
and  environmental  cues  in  a  data  system  format  which 
could  be  translated  into  engineering  performance  spe¬ 
cifications  [40:ii] . 

In  two  additional  reports.  Technology  Incorporated  will  take 
the  STREDS  technique  from  the  theory  of  this  first  report 
to  an  experimental  application  with  an  emerging  W/S. 

STREDS  presents  a  revised  ATD  acquisition  process 
which  will  output  ATD  requirements  in  time  to  meet  IOC  via 
three  phases  (Figure  3-1) . 

The  Initial  Requirements  Format  (IRF)  provides  general 
policies,  training  requirements/objectives,  and  goals.  The 
User  Requirements  Format  (URF)  is  "a  detailed  statement  of 
the  trainer  configuration  which  MAJCOM  desires  to  meet." 

An  Engineering  Design  Format  (EDF)  is  the  specification 
given  to  the  vendor  (40:2).  Of  particular  interest  is  the 
iterative  process  used  to  involve  the  user  and  SIMSPO  in 
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PHASE  III  DTAE  ON  A  REAL  SYSTEM 


STREDS'  ATD 

Acquisition  Process  (40). 


each  phase  of  specification  development. 

LOGICON  Letter.  The  LOGICON  letter  to  the  B-1B 
SPO  is  a  discussion  of  this  company's  training  system  devel¬ 
opment  concepts.  Points  of  interest  are: 

1.  Given  current  ISD  procedures  and  regulations, 
it  is  "a  practical  impossibility"  to  have  a  training  system 
on-line  before  IOC  (7:5). 

2.  Current  training  equipment  design  frequently 
precedes  training  material  development  which  causes  the 
training  program  to  conform  to  equipment  capabilities. 

3.  Pieces  of  training  equipment  are  defined  in 
isolation  from  other  parts  of  the  training  program  "instead 
of  as  an  integrated  subsystem  17:43." 

4.  Training  programs  should  be  designed  as  an 
"Integrated  Training  System"  (TS)  where  all  parts  of  the 
system  are  developed  concurrently  (7:2). 

5.  The  TS  consists  of  the  Training  Management 
System  (TMS)  and  the  Training  Plant  (TP) .  The  TMS  function: 

...  is  to  govern  the  workings  of  the  training  plant 
such  that  the  actual  student  output  . . .  meets  the 
desired  student  output  . . .  and  that  the  actual  resources 
consumed  ...  is  equal  to  or  less  than  the  desired 
resource  consumption  [7:3]. 

The  TP  consists  of  the  components  shown  in  Figure  3-2. 

Its  function  is  to  conduct  and  support  the  training  process. 

6.  "...  the  basic  conceptual  sequence  of  ISD  as 
shown  in  ...  [Figure  3-3]  must  be  preserved  and  followed 
[7:5] 
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TRAINING 


7.  The  iteration  aspect  of  ISD  is  commonly  over¬ 
looked  (7:6). 

8.  LOGICON  summarizes: 

The  point  is,  that  it  is  possible  to  preserve  the 
conceptual  integrity  of  the  ISD  process  and  to  adapt 
it  to  the  time  constraints  imposed  by  the  weapon  system 
procurement  schedule,  if  (and  only  if)  a  top-down  design 
approach  is  used  which  proceeds  in  iterative  steps  from 
a  MACRO  to  a  MICRO  level  of  definition.  The  result  that 
can  be  expected  from  such  a  method  is  an  efficient  and 
integrated  TS  [integrated  training  system]  design  and 
early  implementation  [7:7]. 

9.  For  a  given  training  requirement,  there  are 
"always  numerous  feasible  solutions"  (i.e.,  different  ver¬ 
sions  of  training  systems  and  ATDs) , . . .  "the  optimal  solu¬ 
tion  for  a  given  set  of  training  program  constraints  can 
fairly  easily  be  selected  from  these  alternatives,  once 
they  are  made  explicit  [7:7]."  They  continue: 

Unfortunately  however,  current  ISD  procedures  do 
not  emphasize  the  generation  of  multiple  feasible 
training  system  or  training  system  component  configura¬ 
tions.  The  remedy  is  simply  a  matter  of  requiring  mul¬ 
tiple  configurations  and  a  subsequent  cost-benefit 
analysis.  Contrary  to  intuition,  this  requires  very 
little  extra  effort,  since  training  system  system 
design  should,  according  to  3.2,  proceed  in  stages 
from  MACRO  to  MICRO.  At  the  MACRO  level  it  is  fairly 
easy  to  generate  alternative  solutions  since  one  is 
not  forced  to  deal  with  huge  amounts  of  detail  infor¬ 
mation.  As  decisions  are  made  in  subsequent  stages  the 
number  of  alternatives  gets  constrained  very  rapidly, 
but  always  with  the  reassurance  that  all  feasible 
possibilities  have  been  investigated  and  that  alter¬ 
natives  were  not  eliminated  prematurely  [7:8]. 

10.  Improve  the  probability  of  a  successful  training 
system  by  using  only  "genuine  professional  expertise,"  since 
ISD  cannot  be  reduced  for  the  "non-expert  [7:8].” 
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Hritz  and  Purifoy  Study.  The  Hritz  and  Purifoy 
study  on  Air  Force  maintenance  simulator  design  and  acqui¬ 
sition  includes  items  germane  to  our  research.  They  present 
three  possible  methods  to  compress  the  ISD  process.  They 
introduce  their  solutions  with: 

Throughout  the  project  several  problem  areas  con¬ 
tinually  emerged  ...  Thus,  in  this  section  of  the 
report  these  problem  areas  are  addressed  along  with 
recommendations  and  areas  for  future  research  [20:68]. 

The  report's  first  solution  states: 

It  is  perhaps  reasonable  on  the  surface  to  suggest 
that  more  manpower  be  made  available  to  perform  the 
ISD  analysis  ...  It  is  intuitively  appealing  . . . 
However,  it  should  be  recognized  that  the  ISD  analysis 
is  dependent  upon  the  availability  of  the  data  base 
[20:68-69]. 

Therefore,  extra  people  are  not  useful,  unless  weapon  system 
information  is  available.  Thus,  improved  information  avail¬ 
ability  is  a  possible  solution. 

The  second  solution  suggests  slipping  the  simulator 
delivery  date.  Their  discussion  is  based  on  maintenance 
training,  but  the  concept  may  be  applicable  to  other  train¬ 
ing  situations  such  as  aircrew  training: 

It  has  been  suggested  that  one  way  to  increase  the 
time  available  to  perform  the  ISD  analysis  is  to  use 
actual  equipment  during  the  conversion  training  rather 
than  the  maintenance  trainer.  Using  actual  equipment 
during  the  conversion  training  would  give  the  ISD 
analyst  more  time  to  design  the  maintenance  trainer 
[20:69]  . 

This  solution  is  based  on  three  assumptions.  First,  the 
study  reports  that  seven— level  personnel,  the  skilled 
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technicians,  are  the  first  workers  to  receive  training  in 
new  weapon  systems.  These  skilled  technicians  are  familiar 
with  similar  systems  and  would  not  need  the  simulator. 
Second,  actual,  equipment  would  be  available  for 
training.  Third,  three-level,  apprentice  maintenance  tech¬ 
nicians  are  the  only  trainees  who  require  the  simulator. 
This  suggestion  gains  about  a  year  for  completing  the  ISD 
analysis.  The  report  suggests  further  study  to  determine 
if  seven-level  technicians  could,  indeed,  be  trained  on 
actual  equipment  (20:69-70) . 

The  third  solution  reports  two  methods  to  compress 
the  ISD  process.  The  first  method  would  modify  the  ISD 
process  by  relying  on  more  detailed  behavioral  analysis  of 
the  target  population,  rather  than  skill  and  knowledge 
levels.  The  second  method  would  reduce  the  documentation 
effort  via  word  or  data  processing  (20:70). 

OTHER  ARTICLES/PAPERS.  The  last  source  of  data  is 
articles  and  papers  from  various  publications. 

Douglas  Aircraft.  In  an  article  written  by  members 
of  Douglas  Aircraft  Company's  Human  Factors  Engineering 
function,  an  argument  is  advanced  to  allow  the  prime  air¬ 
frame  contractor  to  conduct  all  ISD  activity.  The  main 
argument  for  this  method  is  early  and  easy  information 
availability: 
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He  [the  contractor]  has  immediate  access  to  all  the 
prerequisite  technical  data  and  documentation.  ISD  can 
be  integrated  with  system  engineering  and  delivery  sche¬ 
dules  for  the  air  vehicle.  His  subject  matter  experts 
(SMEs)  will  be  system  engineers  and  test  pilots  with 
detailed  knowledge  of  system  components  and  operations. 
He  has  access  to  SMEs  from  the  outset,  hence  facilitat¬ 
ing  his  ability  to  complete  early  analysis  of  system 
requirements.  These  capabilities,  combined  with  the 
requirements  for  a  total  systems  approach,  logically 
argues  that  the  system  and  all  its  constituent  elements 
should  be  derived  from  a  common  source  —  the  prime 
contractor  [37:208]. 

The  article  concludes  that  ISD/ATD  development  by  the  prime 
contractor  is  the  best  method  based  on  the  current  weapon 
system  acquisition  environment  (37:207). 

NATO  Report.  NATO  established  the  Advisory  Group 
for  Aerospace  Research  and  Development  (AGARD)  "to  bring 
together  the  leading  personalities  of  the  NATO  nations  in 
the  field  of  science  and  technology  [30:ii]."  Working  Group 
Ten's  objectives  were: 

1.  To  review  the  scope  and  effectiveness  of  current 
simulators. 

2.  To  review  technologies  and  human  behavior  impor¬ 
tant  to  flight  simulation. 

3.  To  identify  areas  needing  increased  research 
(30  :iii)  . 

AGARD  Report  number  159  focuses  attention  on  a  "framework 
for  the  logical  selection  of  training  simulator  facilities 
with  guidance  for  making  the  various  tradeoffs"  and 
"address [es]  the  question  of  how  much  fidelity  is  required 
to  train  a  given  flight  phase  in  isolation  [30:2]." 
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The  report  suggests  a  method  to  develop  the  re¬ 
quirements  for  ATDs  (Figure  3-4) .  The  method  assumes  that 
the  missions  of  an  aircraft  can  be  identified.  Further, 
each  mission  can  be  split  into  flight  phases  (FPs)  .  Once 
you  identify  the  flight  phase,  the  vital  question  becomes 
what  ATD  capabilities  are  needed  to  support  each  FP  (30:2). 


AIRCRAFT  A 


Figure  3-4.  Breakdown  of  W/S  use  into 
Missions  and  Phases. 

The  report  suggests  that  for  each  FP,  the  ATD  capa¬ 
bility  depends  "on  at  least  pilot  experience/background  and 
the  instruction  technique  used  [30:2]."  The  report  con¬ 
tinues  that  there  is  difficulty  in  defining  the  ATD  capabil¬ 
ity  that  is  tied  to  each  FP  because  of  insufficient  knowledge 
of  human  physiology  and  the  fidelity  levels  needed  to  meet 
task  cue  requirements.  To  compensate  for  these  shortcomings, 
the  current  procedure  is  to  simply  specify  the  maximum  or  best 
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ATD  capability  available  (30:2-3).  The  authors  saw  the  lack 
of  research  in  required  fidelity  as  a  major  roadblock  to  the 
ATD  requirements  process. 

Among  the  recommendations /conclusions  presented  are: 

1.  Training  objectives  should  drive  ATD  design  re¬ 
quirements  (i.e.,  the  tasks  or  FPs  the  ATD  is  intended  to 
train) . 

2.  Fidelity  requirements  are  also  dependent  on  the 
training  objectives. 

3.  Cue  requirements  for  specific  training  objec¬ 
tives  require  additional  research. 

4.  Additional  research  is  needed  in  several  areas 
including  the  procedure  to  bridge  the  gap  between  convert¬ 
ing  training  objectives  into  ATD  capabilities  (30:11-12). 

Carlucci  Pre-Planned  Product  Improvement  (P3I)  Letter. 
This  change  to  the  acquisition  process  set  forth  by  Frank 
C.  Carlucci,  The  Deputy  Secretary  of  Defense  is  reviewed 
because  it : 

1.  represents  the  future  environment  in  which  ATD 
acquisition  would  take  place.  P3I  references  are  added  to 
DOD  Directive  5000.1,  Major  System  Acquisition  (49:2). 

2.  presents  an  alternative  ATD  acquisition  strat¬ 
egy  for  consideration. 

In  his  July  6,  1981  letter  entitled  Improved  the  Acquisi¬ 
tion  Process  Through  Pre-Planned  Product  Improvements, 
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Deputy  Secretary  Carlucci  "directed  an  evolutionary  and 
lower  technological  risk  concept  of  ...  P3I  be  implemented 
as  a  means  of  reducing  unit  costs  and  decreasing  acquisi¬ 
tion  time  [3:1]."  We  briefly  review  the  I  process  here. 

P3I  is  a  modification  to  current  acquisition  stra¬ 
tegy.  Where  current  acquisition  strategy  tries  to  have  a 
full-capability  system  designed  for  production,  P3i  uses 
"the  orderly,  time  phased  introduction  of  incremental 
system  capability  to  accommodate  projected  changes  in 
threat  or  to  reduce  risk  in  initial  fielding  of  the  system 
[3:3]."  Specifically,  P3I  has  initial  units  contracted  ' 
which  have  only  partial  capability,  but  with  provisions 
for  future  modifications  which  would  increase  unit  capa¬ 
bility  as  the  need  and  the  technology  become  clarified 
(3:2)  . 

The  objectives  of  P3I  are: 

-shorten  the  acquisition  and  deployment  time  for  a 
new  system  or  an  incremental  capability; 

-reduce  overall  acquisition  and  operating  and 
support  costs; 

-extend  useful  life  of  equipment; 

-combat  military  obsolescence; 

-accomplish  orderly  growth  from  initial  to  mature 
system  reliability;  and 

-reduce  logistics  and  support  problems  entailed 
with  new  material  introduction  [3:2]. 
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While  P  I  is  still  in  its  infancy,  the  concept  may 
be  of  use  in  our  research  project. 

Schilling's  Article.  An  article  written  by  David 
Schj lling  presents  a  planning  methodology  which  may  comple¬ 
ment  P^I.  He  presents  a  strategic  facility  planning  method¬ 
ology  that  enables  decision  makers  to  delay  decisions  on 
final  facility  configurations  for  facility  systems  which 
require  several  years  to  complete.  The  decision  maker 
defines  alternative  facility  configurations  based  on  a 
number  of  possible  future  environments.  These  options  are 
defined  in  a  way  that  maximizes  the  number  of  common  facil¬ 
ities  between  options.  The  common  facilities  are  built 
first.  Then,  as  additional  information  becomes  available, 
the  decision  maker  selects  the  correct  configuration  for 
the  most  likely  future  environment.  Schilling  calls  this 
process  strategic  scenario  planning. 

This  concept  can  be  useful  for  the  acquisition  of 
ATDs  for  emerging  weapon  systems .  Instead  of  alternative 
facility  configurations,  the  acquisition/training  community 
identified  alternative  training  systems  based  on  foresee¬ 
able  weapon  system  configurations  and  operations,  mainten¬ 
ance,  and  training  concepts.  The  idea  is  to  define  ATD 
capabilities  to  be  incorporated  into  the  initial  ATD  de¬ 
signs  and  delay  decisions  on  some  subsystems,  which  are 
subject  to  the  greatest  uncertainty,  until  more  information 
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becames/available.  If  P3I  becomes  a  viable  alternative  for 
ATD  procurement,  then  the  additional  ATD  subsystems  are 
added  as  the  information  becomes  more  certain  (33:1-14). 

Generic  Data  Bases  (GDBs) .  We  review  generic  data 
bases  because: 

1.  NTEC  conducted  research  over  several  years  and 
feels  GDB  has  some  utility. 

2.  ASD/SIMSPO  expressed  interest  in  GDB  use. 

3.  Current  literature  states  that  GDB  use  provides 
real  advantages  during  ISD-type  analysis  in  an  environment 
where  analysis  is  so  constrained  by  time  or  money  that  a 
traditional  ISD  task  analysis  is  not  yet  possible. 

A  well-designed  GDB  can  place  training  system  dev¬ 
elopment  years  ahead  of  a  traditional  ISD  development 
approach.  Traditional  ISD  task  analysis  assumes  that  every 
new  system  is  unique;  therefore,  there  is  no  prior  task 
knowledge;  and  a  complete  task  analysis  is  essential.  GDB, 
on  the  other  hand,  assumes  that  if  there  are  simularities 
or  links  between  existing  and  future  aircraft,  and  if  you 
identify  the  links;  you  can  develop  a  generic  task  list 
without  a  formal  task  analysis  and  complete  up  to  75%  of 
the  project's  task  analysis  before  the  first  aircraft  is 
built  (36:298).  Another  estimate,  attributed  to  ASD  engi¬ 
neers,  claims  a  GDB  task  capability  of  up  to  95%;  we  feel 
this  may  be  optimistic  (4) . 

Mulligan  and  Funaro  of  NTEC  envision  an  inter- 
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service  information  system  with  a  computerized  GDB  which 
includes  information  about  aircraft,  missions,  tasks,  etc. 
They  suggest  three  GDB  versions.  The  first  includes  only 
a  generic  task  listing;  the  second  adds  generic  behavioral 
objectives  to  each  task;  and  the  third  adds  generic  media 
data  to  the  generic  task  and  behavioral  objectives.  They 
argue  that  as  the  data  base  increases  in  capability,  the 
specificity  of  the  outputs  increases  to  the  point  where  the 
ATD  capabilities  are  produced  relatively  quickly  (29:23-40). 

Smith  and  Murray  also  discuss  generic  training 
development  in  their  report  on  Computer  Aided  System  for 
the  Development  of  Aircrew  Training  (CASDAT) .  They  feel 
that  their  generic  and  computer  approach  offers  "signi¬ 
ficant  potential"  over  more  traditional  approaches — even 
those  which  used  data  or  word  processing,  primarily  be¬ 
cause  of  its  generic  aspect.  Their  model  provides  com¬ 
puter  and  generic  aid  to  each  step  of  ISD-type  training 
system  development.  Their  steps  are^ 

1.  Task  analysis 

2.  Behavioral  objectives 

3.  Syllabus  and  media  selection 

4.  Lesson  specification 

The  building  of  the  task  list  is  of  special  interest. 

Smith  and  Murray  claim  that  their  process  can 
identify  about  75%  of  all  operator  tasks.  They  start 
their  process  by  building  a  data  base  of  current  aircraft 
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task  statements  which  are  reduced  to  their  fundamental 
parts.  Once  the  data  base  is  developed,  "common  task 
structures"  for  "all  types  of  aircraft"  are  identified. 

The  authors  continue  task  refinement  by  using  additional 
aircraft  dimensions  such  as  mission  and  weapons  configur¬ 
ation  to  identify  ever  more  specific  task  statements 
(36:297-302) . 

SECTION  3:  PRELIMINARY  CONCLUSIONS 

In  this  section  we  compare  the  items  of  the  liter¬ 
ature/research  review  with  the  criteria  stated  in  Section  1 
and  make  initial  conclusions.  We  summarize  the  comparison 
results  in  Table  3-1  and  table  notes.  Because  the  subject 
matter  is  non-quantitative ,  we  rely  on  our  judgement  for 
the  comparisons.  Clarifying  information/comments  are  pro¬ 
vided  in  the  notes  to  the  table.  The  key  to  the  table's 
symbols  are: 

1.  Plus(+):  In  the  authors'  judgement,  the 
method  clearly  exceeds  the  current  Air  Force  method  (ISD) 
for  that  criterion. 

2.  Minus (-):  In  the  authors'  judgement,  the 
current  Air  Force  method  clearly  exceeds  that  method  for 
that  criterion. 

3.  Zero  (0):  In  the  authors'  judgement,  the 
article  and  current  Air  Force  method  are  comparable  for 
that  criterion. 

60 


4.  Blank {  ) :  The  article  does  not  address  or  does 
not  provide  sufficient  data  for  that  criterion. 


61 


NOTE«  Explanatory  notes  follow. 


Table  3-1 • 

Summary  of  Comparison  with  Criteria. 
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Shorten  analysis 


^While  Army  Aviation  does  have  a  rational  and  meth¬ 
odical  ATD  selection  process  based  on  ISD,  "real  world" 
considerations  (for  example,  see  Hofer)  reduce  its  effec¬ 
tiveness  to  no  better  than  current  Air  Force  acquisition. 

The  major  negative  factor  is  late  ATD  delivery  dates. 

2 

The  Navy's  process  has  two  strengths  worth  noting. 
First,  the  iterative  review  of  the  developing  functional/ 
physical  item  description  by  ISD  and  user  personnel  reduces 
the  probability  of  a  disjoint  between  the  item  that  is 
finally  delivered  and  the  device  the  training  program  needs. 
Second,  the  review  process  is  facilitated  by  the  colloca¬ 
tion  of  ISD  personnel  and  project  engineers. 

3 

The  Cream,  Eggemeier  and  Klein  article  provides 
numerous,  excellent  ideas  which,  if  operationalized,  could 
be  better  than  current  ISD  procedures.  Our  only  negative 
comments  are  the  scant  attention  given  to  emerging  weapon 
systems  and  what  appears  to  be  the  long  time  to  accomplish 
the  analysis. 

4 

The  advantages  of  the  ARI  SOW  are  that  it  presents 
a  rational  method  for  building  an  initial  task  analysis  very 
early  in  the  weapon  systems'  development  and  it  could  im¬ 
prove  the  probability  of  an  ATD  delivery  by  IOC.  This  pro¬ 
cess  is  essentially  ISD  with  an  artificial  task  analysis. 

The  result  of  ARI's  efforts  will  not  be  known  until  early 
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1983,  but  the  effort  is  noteworthy 


5The  Prophet  article  is  included  not  as  a  possible 
solution  source,  but  because  it  cogently  states  ATD  acqui¬ 
sition  problems  as  of  1966  which  have  yet  to  be  solved. 

Sfe  note  that  STREDS  is  only  the  first  of  three 
reports  and,  as  such,  many  of  its  assertions  are  not  sup¬ 
ported.  The  most  positive  aspect  of  the  report  is  their 
proposed  use  of  a  generic  data  base.  If  successfully 
implemented,  the  ATD  generic  data  base  could  greatly 
streamline  the  information  gathering  process  early  in  a 
weapon  system's  development,  and,  if  correctly  constructed, 
aid  in  the  ATD  capability  identification  process.  Our 
initial  concern  that  the  MAJCOM  was  to  have  "operational 
control"  of  the  acquisition  process  was  alleviated  after 
consultation  with  the  ASD  Technical  Consultant  for  this 
report  contract.  The  correct  interpretation  is  that  the 
MAJCOM  would  be  responsible  for  requirements  determination 
with  SIMSPO  assistance  (4) .  To  some  extent,  we  doubt  the 
MAJCOM' s  ability  to  lead  this  effort  because  they: 

1.  would  require  detailed  directions  from  SIMSPO 
to  carry  out  their  duties  (40:36). 

2.  do  not  have  the  necessary  ISD  and  ATD  acquisi¬ 
tion  experience  (40:11). 

3.  have  not  been  able  to  satisfactorily  specify 
ATD  configurations  in  the  past  (40:61). 
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We  realize;  however,  that  the  MAJCOM  must  be  part  of  the 
acquisition  team  because  they  are  the  user  and  a  vital 
source  of  operational  experience. 

7Based  on  the  assumption  that  their  concepts  can 
be  operationalized,  LOGICON's  suggestions  surpass  current 
procedures  for  several  of  our  criteria. 

O 

Hritz  and  Purifoy's  comments  provide  specific 
recommendations  that  addressed  only  two  of  our  six  cri¬ 
teria.  Their  recommendations  are  intuitive,  but  the 
authors  do  not  provide  empirical  support  for  the  ideas 
because  they  are  researching  another  topic,  and  the 
recommendations  are  just  a  byproduct. 

9 

We  doubt  the  Douglas  article's  contention  that 
the  entire  ISD  effort  should  be  contracted  to  the  prime 
airframe  builder  as  biased;  however,  it  will  be  benefi¬ 
cial  to  have  the  easy  access  to  the  data  that  is  described 
in  the  article.  We  feel  this  can  be  accomplished  via  con¬ 
tract  with  the  prime  contractor. 

10The  most  useful  aspect  of  the  NATO  report  is  the 
identification  of  a  methodology  that  enables  task  break¬ 
down  early  in  the  weapon  system's  acquisition  program.  The 
report  quickly  reaches  the  brick  wall  of  identifying 
fidelity  requirements.  The  authors  acknowledge  the  problem, 
but  failed  to  attack  the  issue. 
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11  3 

The  P  I  concept  has  several  positive  attributes. 

By  delaying  decisions  on  certain  ATD  capabilities,  the  de¬ 
signer/developer  gains  time  to  acquire  information.  Second, 
the  use  of  successive  versions  of  the  ATD  allow  you  to  iter¬ 
ate  the  ATD's  capabilities  from  low  to  high  capability/f i- 
delity.  Similarly,  initial  IOC  delivery  can  be  met  with  an 
initial  version  of  the  ATD.  Later  ATD  modifications  can 
reflect  improved  and  clarified  information  about  the  weapon 
system,  technology,  and  ATD. 

12 

Schilling's  strategic  planning  concepts,  in  light 
of  the  P3I  concept,  could  lead  to  a  system  that  would  de¬ 
liver  a  useful  training  device  by  IOC.  A  methodology  still 
needs  to  be  developed  that  would  take  these  ideas  from 
theory  to  application. 

13GDB  has  the  potential  of  being  a  powerful  tool  in 
training  system  development.  The  only  major  question  is 
the  quality  of  the  data  base.  Our  concern  is  that  if 
current  aircrafts'  task  lists  and  data  are  not  correct, 
those  errors  will  be  perpetuated  in  the  GDB.  Our  two 
assumptions  are: 

1.  Mature  weapon  systems  have  correct  task  lists. 

2.  Generic  behavioral  objectives  and  media  data 
are  reviewed,  standardized,  and  improved  as  part  of  the 
GDB  building  process. 
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After  this  review,  we  concluded  that  none  of  the 
methods  met  all  of  our  criteria  but  that  many  good  ideas 
exist.  Many  of  these  concepts,  if  implemented,  would 
alleviate  the  problem;  however,  no  documented  attempt  has 
been  made  to  infuse  many  of  these  ideas  into  the  Air  Force 
ATD  acquisition  process.  We  proceed  to  formulate  a  model/ 
process  of  what  would  be  a  better,  more  realistic  process 
that  synthesizes  many  of  the  ideas  presented  in  the  liter¬ 
ature  and  some  new  ideas  (based  on  experience,  and  systems 
and  management  concepts) .  This  model  is  developed  and  pre¬ 
sented  in  the  next  chapter. 
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CHAPTER  4 


MODEL  DEVELOPMENT 

Chapter  4  covers  phases  five  (Solution  Overview) 
and  six  (Detailed  Design)  of  our  modified  Weinberg  meth¬ 
odology.  Section  1  (Phase  Five — Solution  Overview)  is  a 
general  discussion  of  our  model  features  and  their  impact, 
while  Section  2  (Phase  Six — Detailed  Design)  is  a  detailed 
Data  Flow  Diagram  (DFD)  and  narrative  of  our  Training  Sys- 
tem/ATD  Development  Model.  Many  of  the  ideas  in  these 
sections  are  taken  from  the  literature  reviewed  in  Chapter 
3  (Figure  4-1) . 

SECTION  1;  PHASE  FIVE— SOLUTION  OVERVIEW 

As  shown  in  Chapter  1,  the  current  Air  Force  ATD 
acquisition  process  is  inadequate  for  emerging  W/S  training 
developments  and  none  of  the  systems  reviewed  in  Chapter  3 
met  our  criteria.  However,  a  careful  mixture  of  several 
ideas  will  lead  to  what  the  authors  believe  is  an  improved 
ATD  acquisition  process.  We  group  the  features  of  our  im¬ 
proved  process  into  four  areas: 

1.  Management  and  personnel, 

2.  Information  availability, 

3.  Contracting  and  delivery  strategies, 

4.  Training  System/ATD  Development  Model. 

MANAGEMENT  AND  PERSONNEL.  There  are  four  manage¬ 
ment  and  personnel  features: 
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FEATURE 


SOURCE 


Generic  Data  Base 

Access  to  Prime  Contractor 
Data 

Mission  Analysis 

MACRO  to  MICRO  analysis  via 
iterative  process 

Collocated/designated  ATD 
design  team 

Cost-Benefit  Analysis 

ISD  process  modifications 
Scenario  planning/P^I 

Specifications 

ASD  SIMSPO  role 


STREDS  (40) 

Mulligan  and  Funaro  (29) 
Coward  (4) 

Smith  and  Murray  (36) 
Douglas  Aircraft  (37) 


AGARD  NATO  (30) 

ARI  (41) 

NTEC  (51) 

NTEC  (50) 

LOGICON  (7) 

NTEC  (50) 

Cream  (6) 

LOGICON  (7) 

STREDS  (40) 

Mulligan  and  Funaro  (29) 
DOD  Econ  Handbook  (9) 
Cream  ( 6 ) 

Hritz  and  Purifoy  (20) 


Carlucci  (3) 

DOD  Directive  5000.1  (49) 
Schilling  (33) 

Authors 
Gainer  (14) 

Coward  (4  and  5) 

AGARD  NATO  (30) 

STREDS  (40) 


Figure  4-1.  Features  and  Source (s). 
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1.  Centralize  process  control/direction, 

2.  Improve  quality  and  training  of  personnel, 

3.  Use  team  concept) 

4.  Use  collocation  or  improve  access  between  team 

members. 

Centralize  the  control  and  direction  of  early  W/S 
training  and  ATD  development.  While  a  Navy  plan  to  central¬ 
ize  all  DOD  ATD  acquisition  at  one  location  may  be  polit¬ 
ically  unacceptable,  the  Air  Force  must  organize  a  central¬ 
ized  staff  function — a  Training  SPO  (TSPO) — responsible  for 
aiding  the  MAJCOMs  in  initial  training  system  development 
and  conducting  ATD  acquisition.  The  SIMSPO  would  become 
the  nucleus  of  the  new  TSPO.  Additional  instructional 
development,  engineering,  operations,  and  research  exper¬ 
tise  would  be  added.  After  organizing,  the  unit  would  pre¬ 
pare  materials  and  procedures  specifically  designed  for 
emerging  W/S  training  systems  and  ATDs  (such  as  a  GDB) . 

When  a  new  W/S  SPO  came  into  being,  the  TSPO,  under  the 
command  of  a  single  manager,  would  be  prepared  to  direct 
the  training  system  and  ATD  development  for  the  MAJCOMs 
and  W/S  PM. 

Man  the  highly  technical  engineering  and  instruc¬ 
tional  development  positions  with  Government  Service  (GS) 
personnel.  Using  GS  workers  allows  for  more  long-term  and 
stable  assignments,  justifies  the  expenditures  for  long¬ 
term  training  or  education  for  workers,  allows  recruitment 
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of  highly  trained  professionals,  and  retains  corporate 
knowledge.  The  role  of  military  members  should  be  in  the 
areas  of  command,  support,  Subject  Matter  Experts  (SMEs) , 
and  program  management. 

Use  the  team  concept  for  training  system  and  ATD 
development.  Organize  a  program  team  consisting  of  the 
following: 

1.  Program  managers  (TSPO  and  user  command), 

2.  Early  training  system  developers  (TSPO)  and 
ISD  specialists  (TSPO,  user  command,  and  prime  airframe 
contractor) , 

3.  Design  and  human  factors  engineers  (TSPO), 

4.  ILS  managers  (TSPO), 

5.  SMEs  (W/S  user  cdmmand  and  prime  airframe  con¬ 
tractor)  , 

6.  W/S  SPO, 

7.  ATD  contractor. 

The  amount  of  time  a  team  member  devoted  to  the 
project  would  vary  depending  on  the  stage  of  the  program 
and  his  function.  For  example,  the  W/S  SPO  members  would 
probably  only  work  part-time  during  the  entire  develop¬ 
ment  and  the  ATD  contractor  could  not  participate  at  all 
until  the  ATD  production  contract  was  awarded . 

There  are  four  advantages  to  the  team  concept. 
First,  it  insures  that  the  multiple  perspectives,  desires, 
and  needs  of  the  various  parties  are  represented.  Second, 


it  aids  in  the  coordination  and  integration  of  all  activi¬ 
ties  occurring  in  the  dynamic  environment  of  major  W/S, 
training,  and  ATD  development  projects.  Thirl,  it  mini¬ 
mizes  organizationally  induced  adversarial  relationships. 
Fourth,  it  encourages  the  free  flow  of  information  between 
team  members . 

Collocate  the  members  of  the  development  team  for 
three  reasons.  First,  it  encourages  the  day-to-day  inter¬ 
action  necessary  to  implement  the  macro-to-micro  and  itera¬ 
tive  aspects  of  this  system.  The  development  of  each  ver¬ 
sion  of  the  training  system  (iteration)  involves  daily 
interaction  with  team  members;  and,  each  time  a  version  is 
complete,  the  team  members  and  management  perform  a  review. 
These  activities  and  others  require  that  the  team  be  collo¬ 
cated. 

Second,  the  ATD  must  evolve  concurrently  with  other 
training  elements.  The  ATD  is  only  one  part  of  the  larger 
training  system  and  if  it  does  not  develop  in  consonance 
with  the  whole,  the  delivered  ATD  could  have  very  little 
in  common  with  the  other  subsystems  and  be  virtually  'se- 
less.  To  prevent  this  disjoint,  daily  interaction  with 
the  other  subsystems  is  needed,  and  this  can  only  be  accom¬ 
plished  through  collocated  activities. 

Third,  collocated  functions  foster  an  atmosphere 
that  will  lead  to  a  free  exchange  of  ideas  not  only  on  the 
training/ATD  program,  but  also  on  such  things  as  ISD  and 
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GDB  improvements  and  developing  training  technology  and  its 
applications.  This  cadre  will  also  maintain  a  standardized 
training  development  process  to  be  used  for  emerging  W/Ss. 

If  collocation  is  not  possible,  then  a  real-time, 
interactive  computer  system  is  needed  to  provide  the  access 
between  team  members  required  to  accomplish  the  first  two 
activities.  In  this  way,  the  team  members  can  have  access 
to  the  developing  project  data  base  and  the  GDB.  Team 
members  will  be  able  to  communicate  changes  and  new  infor¬ 
mation  to  each  other  despite  being  located  in  different 
geographical  areas. 

INFORMATION  AVAILABILITY.  There  are  two  ways  to 
provide  better,  more  timely  information: 

1.  More  accessibility  to  prime  contractor  infor¬ 
mation  earlier, 

2.  Generic  Data  Base  and  Mission  Analysis. 

Improve  access  to  the  prime  contractor's  W/S  infor¬ 
mation  via  W/S  contract  data  items.  As  a  member  of  the 
development  team,  the  prime  contractor's  representatives 
would  act  as  purveyors  of  current  W/S  information  to  the 
team.  In  addition,  the  team  would  have  consultation  rights 
to  the  prime's  development  contracts  with  accompanying 
cost  increases.  The  amount  of  prime  provided  information 
would  be  most  important  early  in  the  W/S's  development  and 
decrease  in  importance  later  as  more  detailed  information 
becomes  available  via  Air  Force  sources .  Two  other  ways  to 


provide  early  information  are  Mission  Analysis  and  GDBs. 

Develop  and  use  Mission  Analysis  and  an  inter¬ 
service  Generic  Data  Base  for  early  training  system  and  ATD 
development.  Mission  Analysis  and  GDB  enable  the  training 
developer  to  quickly  generate  much  of  the  information  needed 
to  support  early  training  and  ATD  decisions.  The  Mission 
Analysis,  as  discussed  in  Chapter  3  as  part  of  the  ARI  and 
NATO  studies,  along  with  the  prime  contractor's  estimate 
of  emergency  and  operator  maintenance-type  tasks  yields  an 
initial  operator  tasks  list  (TL) .  At  the  same  time  interro¬ 
gations  of  the  GDB  yield  the  common  TL;  behavioral  objec¬ 
tives  (BOs) ;  criticality,  frequency  and  difficulty  of  per¬ 
formance  (C/F/D)  scores;  and  media  data  (MD) .  A  comparison 
of  Mission  Analysis  and  GDB  TLs  identifies  unique  and  new 
tasks.  The  effort-'  by  the  training  developers  are  then 
directed  toward  developing  estimates  of  BOs,  C/F/D  scores, 
and  MDs  for  the  unique  tasks. 

The  features  and  structure  of  the  GDB  determine  its 
utility.  The  features  of  Mulligan  and  Funaro's  most  complex 
GDB  should  be  used  for  the  inter-service  data  base.  That 
data  base  includes  TL,  BO,  and  MD  data.  This  structure 
allows  the  training  developer  to  quickly  develop  relatively 
fine  "specificity  of  instructional  regimens  [46:Fig.l5] . " 

While  the  utility  and  general  operation  of  GDBs  are 
documented  in  the  training  literature,  we  found  little  de¬ 
tailed  information  about  the  GDB's  structure.  However,  we 
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feel  that  each  data  base  entry  should  be  an  identified  task 
from  a  past  W/S  which  includes  five  fields  of  information 
(Figure  4-2) .  The  first  entry  is  key  words  which  describe 
the  task,  aircraft,  and  equipment  in  sufficient  detail  to 
allow  searches  and  commonality  sorts  of  the  data  like  that 
discussed  by  Smith  and  Murray.  The  second  entry  is  the  task 
name.  The  third  entry  is  the  behavioral  objective  used  to 
measure  tne  trainee's  progress  (i.e..  When  the  trainee  per¬ 
forms  the  behavioral  objective,  we  are  satisfied  that  he 
knows  how  to  perform  the  task) .  The  fourth  entry  is  the 
tasks'  criticality,  frequency,  and  difficulty  of  performance 
(C/F/D)  ratings;  these  are  subjective  ratings  developed  by 
the  SMEs  and  training  developers.  The  fifth  area,  MD,  con¬ 
sists  of  a  media  mix  code  (what  media  are  most  appropriate) 
and  capability  scale  (used  to  define  the  more  complex  media, 
i.e.,  ATDs) .  MD  is  the  most  complex  and  uncertain  area; 
but  it  has  the  potential  of  becoming  a  powerful  tool  in  ATD 
acquisition. 

Since  there  is  little  agreement  or  useable  informa¬ 
tion  on  MD,  we  have  established  our  own  propositions. 

First,  we  propose  that  MD's  purpose  is  to  help  select  the 
correct  media  mix  to  train  a  particular  task  and  to  communi¬ 
cate  to  the  engineer  the  capabilities  needed  to  support  that 
task's  training.  Second,  while  there  are  ISD  decision  rules 
to  help  in  media  mix  selection,  we  propose  that  the  inter¬ 
service  GDB  developers  must  undertake  the  effort  to  develop 
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Y-AXIS 


Figure  4-2.  Generic  Data  Base  Format, 
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the  capability  scales  and  to  conduct  a  task-by-task  MD 
analysis.  Third,  we  propose  that  the  capability  scales  will 
probably  go  through  several  versions  of  increasing  complex¬ 
ity  and  utility. 

While  the  detailed  development  of  the  MD  capability 
scales  is  not  of  interest  to  this  research  effort,  we  will 
discuss  the  topic  on  a  macro  level.  The  initial  capability 
scale  will  be  an  enumeration  of  those  cues  that  the  student 
must  receive  in  the  ATD.  The  NATO  AGARD  study  suggests  four 
categories  of  cues: 

1.  Cockpit  display  systems, 

2.  External  visual  scene, 

3.  Cockpit  motion  system, 

4.  Verbal  and  nonverbal  audio  environment 
00  :Fig. 4) • 

The  capability  scale,  then,  consists  of  the  cues  necessary 
to  provide  those  categories.  For  each  cue,  a  Yes-No  or 
degree  (fidelity)  scale  is  developed.  For  example,  a  cue 
in  the  cockpit  motion  category  would  be  degrees  of  freedom 
(i.e.,  how  many  axis  of  motion  are  needed).  The  degrees  for 
that  cue  would  range  from  zero  (no  motion)  to  three  (motion 
on  all  three  axes) . 

The  ultimate  capability  scale  builds  on  the  earlier 
scales.  In  this  version,  the  cue  data  is  tied  to  product 
specif ication (s)  needed  to  have  an  ATD  developed  which  pro¬ 
vides  the  needed  cues.  Again,  using  degrees  of  freedom  as 
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an  example ,  zero  degrees  of  freedom  might  call  up  a  specifi¬ 
cation  for  a  stationary  cockpit  platform,  while  the  three 
degrees  of  freedom  might  call  up  specifications  related  to 
hydraulic  pumps,  related  equipment,  and  pump  service  con¬ 
tracts.  The  use  of  the  MD  is  relatively  simple,  but  the 
results  would  be  powerful. 

Regardless  of  the  version  of  MD  developed,  the  use 
is  the  same  in  the  ATD  requirements  process.  After  using 
the  media  mix  codes  to  identify  a  group  of  tasks  normally 
trained  in  some  type  of  ATD,  the  job  of  identifying  the 
ATD's  capability  begins.  As  the  list  of  tasks  to  be  trained 
in  an  ATD  configuration  is  compiled,  a  large  matrix  is 
formed.  The  X-axis  of  the  matrix  is  the  list  of  tasks  to  be 
trained  in  that  ATD  configuration.  The  Y-axis  of  the  matrix 
in  the  MD  portion  of  the  data  base  is  a  list  of  ATD  cues  and 
the  required  fidelity.  Each  column,  then,  represents  the 
cues  and  their  fidelity  necessary  to  train  a  specific  task. 

A  summation  along  the  X-axis  indicates  the  ATD  cue  and 
fidelity  needed  to  train  the  list  of  tasks.  With  this  data, 
the  engineer  would  develop  the  product  specifications  based 
on  the  MD  data  plus  his  ability.  However,  if  we  are  using 
the  ultimate  capability  scale,  the  product  specifications 
would  be  keyed  to  the  cue  and  fidelity  responses.  So  a 
summation  on  the  Y-axis  indicates  the  specifications  needed 
to  support  each  task  in  isolation,  while  a  summation  on  the 
X-axis  indicates  the  specifications  needed  to  support  the 


entire  list  of  tasks.  The  engineer's  job  becomes  one  of 
reviewing  and  correcting  the  product  specification  output. 

The  power  of  this  process  comes  from  our  ability 
(with  the  aid  of  a  computer)  to  quickly  mix  and  match 
various  combinations  of  tasks.  By  adding  and  deleting 
tasks,  we  change  the  needed  capability,  use,  and  cost  of  the 
ATD  configuration.  When  teamed  with  an  iterative  cost-bene¬ 
fit  analysis,  we  can  easily  generate  numerous  cost/capabil- 
ity/tasks-trainable  combinations  for  consideration. 

There  are  eight  steps  in  the  GDB  effort.  As  noted 
by  Smith  and  Murray,  the  first  step  is  to  gather  the  task 
data  from  all  of  the  services'  aircrew  training  programs  and 
to  normalize  (remove  writer  peculiarities  and  put  in  common 
formats)  the  task  titles/statements.  This  makes  the  tasks 
compatible.  Second,  the  word  keys  are  developed  which  allow 
additional  dimensions  of  each  task  to  be  accessed 
(36:297-298).  Third,  the  C/F/D  ratings  are  established  by 
SMEs  and  training  developers.  Fourth,  the  tasks'  BO  are 
reviewed  and  normalized  for  GDB  use.  Fifth,  the  MD  is  dev¬ 
eloped.  Sixth,  data  base  management  and  user  programs  are 
written.  Seventh,  all  data  is  entered  into  the  computer. 
Eighth,  the  process  is  tested  and  validated. 

We  close  the  GDB  discussion  with  a  caveat  to  the 
reader,  the  areas  of  MD  and  cue  requirements  are  ill-de¬ 
fined  and  in  need  of  additional  theoretical  and  applied 
research. 
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CONTRACTING  AND  DELIVERY  STRATEGIES.  While  the 


thrust  of  our  research  has  been  in  training  and  ATD  devel¬ 
opment,  three  contracting  and  delivery  strategies  give  the 
developers  more  time  to  accomplish  their  actions.  The 
features  are: 

1.  Scenario  development, 

2.  P3It 

3.  Use  actual  equipment  or  reduced  fidelity  ATDs 
for  early  operator  training. 

Use  strategic  scenario  planning  in  the  development 
of  ATDs  and  their  production  contracts.  Strategic  scenario 
planning  is  an  important  input  into  the  design  of  alternate 
ATD  configurations.  The  idea  is  to  include  tasks  with  low 
risk  MRs  and  ATD  technology  in  early  ATD  configurations  and 
omit  high  risk  tasks.  As  the  omitted  tasks'  MRs  and  tech¬ 
nological  risk  decreases,  add  these  capabilities.  The 
result  of  this  process  is  a  set  of  ATD  configurations  which 
are  low  risk,  within  current  technology,  but  are  initially 
limited  in  their  use  (i.e.,  some  tasks  requiring  ATD 
support  are  not  covered  by  the  ATD  due  to  excessive  risk) . 
Additional  capability  comes  in  later  iterations  through  the 
Training  System/ATD  Development  Model  either  before  or 
after  basic  ATD  design  freeze.  The  new  capabilities  result 
in  increased  instructional  capability.  This  incremental 
design  process  requires  incremental  contracting  and  deliv¬ 
ery. 
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P^I  provides  this  incremental  contracting  and  deliv¬ 
ery.  It  is  unlikely  that  a  training  system,  the  ATD 

design,  and  ATD  technology  will  ever  develop  fast  enough 

3 

to  provide- a  fully  capable  ATD  at  W/S  delivery.  P  I  accom¬ 
modates  this  problem  by  allowing  for  several  increasingly 
capable  versions  of  the  ATD.  The  initial  production  ATD 
would  be  a  basic  model  (ATD-Version  A  or  ATD-A)  with  only 
low  risk  capabilities.  The  ATD-A  would  be  designed  with  the 
idea  that  modifications  or  modules  will  be  added  later  to 
incorporate  additional  capabilities.  The  reader  is  reminded 
that  iterative  passes  through  the  Training  System/ATD 
Development  Model  would  be  necessary  to  insure  thi  instruc¬ 
tional  system  adapts  to  the  changing  ATD  capability  and  use. 

Use  actual  equipment  or  ATD-As  for  early  training. 
This  feature  takes  into  account  that  having  a  completely 
capable  ATD  at  IOC  may  be  impossible  and  offers  two  alter¬ 
natives.  First,  the  Hritz  and  Purifoy  suggestion  that  actual 
equipment  be  used  for  initial  training  deserves  study.  If 
it  is  proven  that  initial  trainees  in  new  W/Ss  are  the  more 
experienced  crewmembers  transitioning  from  other  aircraft, 
then  the  trainee  may  be  able  to  overcome  the  disadvantage 
of  having  no  ATD  while  less  experienced  crewmembers  may  have 
more  need  for  the  ATD.  If  the  initial  cadre  to  be  trained 
is  predominantly  crewmembers  of  little  experience,  then  the 
second  suggestion  is  to  use  ATD-As  for  training.  While 
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using  the  ATD-A  provides  a  degraded  training  program,  the 
training  program  may  still  be  superior  to  using  actual 
equipment.  The  result  of  both  features  is  the  same — the 
initial  cadre  receives  degraded  training  with  the  promise 
of  an  improved  ATD  and  training  program  later. 

TRAINING  SYSTEM /ATP  DEVELOPMENT  MODEL.  All  of  the 
foregoing  features  operate  within  a  revised  Training  System/ 
ATD  Development  Model.  That  model  is  presented  via  Data 
Flow  Diagrams  (DFDs)  in  the  next  section,  but  inherent  limit¬ 
ations  of  DFDs  require  two  model  features  be  presented  here; 

1.  Macro  to  micro  development 

2.  Iterative  development 

As  presented  by  LOGICON,  the  macro  to  micro  and  iter¬ 
ative  development  processes  offer  a  method  to  "preserve  the 
conceptual  integrity  of  the  ISD  process  and  to  adapt  it  to 
the  time  constraints  imposed  by  the  weapon  system  procure¬ 
ment  schedule  17:7]."  The  macro  to  micro  process  assumes 
that  the  amount  of  information  increases  and  the  quality 
improves  during  the  W/S  development.  Similarly,  the  accu¬ 
racy  and  quality  of  decisions  based  on  that  information  im¬ 
proves.  This  is  fortunate  because  initial,  general  deci¬ 
sions  in  training  and  ATD  development  can  be  made  based  on 
macro,  early  data,  while  the  final  specific  decisions  require 
micro,  later  data. 

When  macro  to  micro  data  is  teamed  with  the  iterative 
process,  a  workable  methodology  appears.  With  repeated  iter- 
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ations  of  an  ISD-based  process,  the  clarity  of  needs  and 
capabilities  increases  as  the  W/S  develops.  The  use  of  the 
GDB,  Mission  Analysis,  and  conventional  ISD  task  analysis 
fits  into  this  methodology.  During  the  macro  stage  of 
development,  the  Training  System/ATD  Development  Model  uses 
and  depends  upon  generic,  simulated,  and  estimated  data. 
Therefore,  the  model's  outputs  allow  the  decision  maker  to 
make  initial,  general  decisions — the  best  possible  decisions 
based  on  current  information.  And,  as  more  detailed  W/S 
information  becomes  available,  the  project  GDB  and  Mission 
Analysis  is  compared  and  updated  with  improved,  more  micro 
information.  Using  the  improved  project  GDB,  the  model 
enables  the  decision  maker  to  make  better,  more  accurate 
decisions  as  the  W/S  development  proceeds.  Over  time  all 
tasks  in  the  project  GDB  are  replaced  with  conventionally 
obtained  ISD  task  analysis,  and  the  resulting  actual  data 
base  can  be  used  with  the  conventional  ISD  model;  however, 
this  would  not  be  possible  until  late  in  the  W/S's  develop¬ 
ment  or  after  IOC. 

SECTION  2:  PHASE  SIX— DETAILED  DESIGN 

The  purpose  of  this  section  is  to  explain  our  pro¬ 
posed  system  in  greater  detail  via  Data  Flow  Diagrams  (DFDs) 
and  a  narrative.  The  features  introduced  in  section  one  are 
imbedded  in  our  proposed  system.  The  DFD  is  three  levels 
deep: 


1.  Level  1  is  what  Weinberg  calls  an  overview  of 
the  entire  system  and  is  shown  on  Figure  4-4  (Figures  4-4 
through  4-11  are  grouped  at  the  end  of  this  chapter  for 
easier  reference) . 

2.  Level  2  is  a  more  detailed  view  of  the  two  pro¬ 
cesses  shown  in  Figure  4-4.  Process  1.0  is  expanded  and 
shown  on  Figure  4-5,  and  process  2.0  is  expanded  and  shown 
on  Figure  4-9. 

3.  Level  3  is  an  even  more  detailed  view  of  each 
process  of  level  2. 

Figure  4-3  shows  the  relationship  of  each  process 
and  figure. 

OVERVIEW  DFD .  Figure  4-4  shows  the  two  major  pro¬ 
cesses  in  our  proposed  model — Develop  Training  System  (1.0) 
and  Design  ATD  Element  (2.0) — along  with  major  inputs  and 
outputs.  In  this  section,  our  definition  of  Training  Sys¬ 
tem  (TS)  is  that  suggested  by  LOGICON  and  reported  in  Chapter 
3  of  our  research. 

1.0  DEVELOP  THE  TRAINING  SYSTEM.  Figure  4-5  shows 
the  three  logical  processes  that  are  performed  in  order  to 
develop  the  TS.  They  are: 

1.  Identify  the  Training  Requirements  (TRs) .  TRs 
are  those  tasks  which  must  be  trained  by  the  TS. 

2.  Develop  alternate  training  systems  (1.2). 

3.  Select  training  system  (1.3). 

Process  1.0  is  iterative.  The  first  pass  through 
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1.0  occurs  early  in  the  W/S  development  and  results  in  ini¬ 
tial  decisions  about  the  TS  and  provides  initial  input  to 
the  ATD  acquisition  team.  As  more  detailed  W/S  informa¬ 
tion  becomes  available,  subsequent  passes  refine  and  im¬ 
prove  the  TS.  The  TS  evolves  from  macro  definition  of  the 
elements  to  more  micro  definitions.  The  pliability  of  the 
TS  is  constrained  over  time  as  more  and  more  TS  elements 
become  fixed.  For  example,  if  a  TS  decision  caused  the 
Air  Force  to  enter  into  a  contract  with  Control  Data  for 
PLATO  computer-based  training,  future  TS  versions  must  treat 
the  PLATO  decision  as  fixed,  unless  there  is  strong  evidence 
to  justify  cancelling  the  contract.  This  iterative  process 
continues  throughout  the  training  development  effort  until 
the  TS  responsibility  is  transferred  from  the  TSPO  team  to 
the  user  for  TS  maintenance. 

Of  special  interest  to  our  research  is  the  ATD's 
element  of  the  TS.  This  element  can  change  with  the  TS 
until  that  time  that  design  freeze  occurs  in  order  to  meet 
IOC  delivery  of  the  particular  ATD.  The  freeze  time  is 
determined  by  the  particular  ATD  and  its  acquisition  pro¬ 
cess.  Relatively  simple  ATDs  like  part  task  trainers  would 
have  late  freeze  dates  while  the  complex  weapon  system 
trainers  (WSTs)  have  early  freeze  dates.  Our  primary  con¬ 
cern  is  with  the  more  complex  ATDs. 

Two  key  outputs  of  1.0  related  to  ATDs  are  the  Vali¬ 
dated  SON  and  the  ATD  Program  Plan.  The  SON  is  the  pre- 
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ferred  method  to  identify  ATD  neeus,  because  it  allows  for 
alternative  solutions  to  the  need.  The  W/S  PMD ,  modified  to 
recognize  an  ATD  requirement,  is  the  second  method  to  docu¬ 
ment  ATD  needs.  The  ATD  Program  Plan  represents  the  plan¬ 
ning  process  that  insures  successful  completion  of  the 
successive  processes  of  this  model.  Two  parts  of  the  plan 
are  the  acquisition  plan  and  the  Integrated  Logistics  Support 
(ILS)  plan.  These  will  guide  the  activities  and  team  mem¬ 
bers  toward  a  timely  completion  of  all  the  activities  in  the 
program. 

1.1  IDENTIFY  TRAINING  REQUIREMENTS.  Figure  4-6 
shows  the  eight  logical  processes  that  occur  in  order  to 
identify  Training  Requirements  (TRs) .  This  phase  equates 
with  the  first  two  steps  of  the  current  ISD  process:  ana¬ 
lyze  system  requirements  (i.e.,  task  analysis)  and  define 
education/training  requirements . 

1.1.1  COLLECT  W/S  INFORMATION  AND  PARAMETERS.  The 
first  activity  is  to  collect  W/S  information  and  parameters. 
This  activity  continues  throughout  the  acquisition  process 
of  the  ATD.  It  must  begin  here  to  provide  the  information 
necessary  for  processes  1.1.2  and  1.1.3.  The  W/S  informa¬ 
tion  is  grouped  into  three  areas.  First,  the  missions  that 
the  new  W/S  will  perform.  Second,  the  functions  that  the 
W/S  must  perform  to  complete  those  missions.  Third,  the 
performance  parameters  (speed,  for  example)  that  will 
enable  mission  accomplishment.  The  missions,  functions, 
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and  performance  parameters  are  used  to  perform  the  common¬ 
ality  analysis  (1.1.2)  and  a  mission  analysis  (1.1.3). 

1.1.2  PERFORM  A  COMMONALITY  ANALYSIS.  A  software 
program  is  used  to  perform  a  commonality  analysis  with  the 
GDB  using  the  key  word  tabs.  The  TSPO  loads  the  W/S  infor¬ 
mation  and  parameters.  Then,  using  this  input,  TSPO  asks 
for  a  task  listing  of  tasks  common  to  the  new  W/S  and  de¬ 
ployed  W/Ss.  Then,  upon  request,  the  GDB  will  provide  other 
data  in  the  tasks'  file. 

1.1.3  PERFORM  A  MISSION  ANALYSIS.  This  process  is 
described  in  Chapter  3.  The  objective  of  mission  analysis 
is  to  use  the  W/S's  missions,  functions,  and  performance 
parameters  to  generate  an  artificial  task  list.  This  list 
includes  all  of  the  tasks  that  are  performed  in  the  W/S. 

This  list,  the  control  TL,  is  different  from  that  in  1.1.2 
since  it  may  include  system  unique  tasks  not  found  in  the 
GDB. 

1.1.4  IDENTIFY  THE  UNIQUE  TASKS.  Now,  compare  the 
control  TL  and  the  generic  TL.  As  a  result  of  this  compari¬ 
son,  the  TSPO  identifies  the  task  list  that  is  unique  to  the 
W/S. 

1.1.5  DEVELOP  ESTIMATE  OF  BO,  C/F/D  AND  MD.  For 
the  unique  tasks,  it  is  now  necessary  to  make  educated 
guesses  about  associated  BO,  C/F/D,  and  MD.  The  guesses  are 
made  using  the  latest  W/S  information  available  and  are 
updated  or  replaced  as  better  information  becomes  available. 
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1.1.6  UPDATE  GDB  FOR  UNIQUE  TL,  BO,  C/F/D ,  AND  MD. 
After  the  unique  TL,  BO,  C/F/D,  and  MD  have  been  identified, 
add  these  to  the  GDB.  This  is  necessary  for  two  reasons. 
First,  it  maintains  and  updates  the  current  system  baseline. 
Second,  after  this  data  is  validated  with  this  W/S,  it  can 
be  used  for  subsequent  W/S  developments. 

1.1.7  BUILD  PROJECT  BASELINE.  The  synthesis  of  the 
generic  and  unique  TLs,  BOs,  C/F/Ds,  and  MDs  is  a  dedicated, 
project  data  base.  We  define  baseline  as  the  information  in 
this  data  base  at  any  given  time. 

1.1.8  DETERMINE  TRAINING  REQUIREMENTS.  This  process 
determines  which  of  the  tasks  in  the  project  baseline  need 

to  be  trained.  By  comparing  what  the  student  population 
knows  before  they  enter  the  TS  with  the  project  baseline 
(what  they  must  know  to  operate  the  W/S) ,  we  identify  the 
tasks  which  need  to  be  taught.  For  example,  if  we  deter¬ 
mine  that  all  students  entering  an  advanced  pilot  training 
program  know  how  to  file  a  flight  plan,  we  delete  that  ac¬ 
tivity  from  the  advanced  training  program;  however,  if  the 
student  population  does  not  know  how  to  perform  a  particu¬ 
lar  task,  we  would  establish  a  TR  for  that  task. 

1.2  DEVELOP  ALTERNATE  TRAINING  SYSTEMS.  Figure 
4-7  shows  three  logical  processes  that  occur  in  order  to 
convert  the  TRs  of  1.1  into  a  set  of  alternative  TSs. 

1.2.1  ENUMERATE  ALTERNATE  TRAINING  STRATEGIES. 

During  this  process  the  training  development  team  is  given 
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the  latitude  to  enumerate  as  many  alternate  training  strat¬ 
egies  as  possible.  Since  there  are  no  imposed  constraints, 
conventional  and  unconventional,  standard  and  nonstandard 
strategies  are  all  given  equal  weight  and  consideration. 

Among  the  strategic  variables  considered  are: 

1.  Intensity  and  sophistication  of  ATDs, 

2.  Degree  of  training  centralization  (i.e.,  one 
large  training  facility  to  many  small  base  level  facilities) , 

3.  Degree  of  computer  aided  instruction  used, 

4.  Self-paced  instruction  vs.  standard,  group 
classroom  presentations. 

1.2.2  DEVELOP  ALTERNATE  TRAINING  SYSTEMS.  In  this 
process,  training  systems  are  developed  to  carry  out  the 
various  strategies  enumerated  in  1.2.1.  The  elements  of  the 
TP  and  TMS  are  considered  and  documented.  In  early  itera¬ 
tions,  develop  the  TSs  at  a  MACRO  level.  In  later  itera¬ 
tions;  however,  as  more  information  is  available  and  fewer 
TSs  are  still  in  consideration,  the  TSs  can  be  more  de¬ 
tailed. 

1.2.3  CONSTRAIN  ALTERNATE  TRAINING  SYSTEMS.  In 
this  process,  the  strategies  and  TSs  of  1.2.1  and  1.2.2  are 
constrained  by  known  constraints.  The  types  of  constraints 
considered  may  include: 

1.  Funding  levels, 

2.  MAJCOM  desires  and  needs, 

3.  W/S  development  schedule, 
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4.  Degree  of  W/S  technical  sophistication, 

5.  Student  population  estimates, 

6.  Instructor  force  ability, 

7.  Existing  facilities, 

8.  ATD  state-of-the-art. 

The  output  of  1.2.3  is  a  set  of  feasible  TSs  which  are  can¬ 
didates  for  selection. 

1.3  SELECT  THE  TRAINING  SYSTEM.  Figure  4-8  illus¬ 
trates  the  six  processes  that  lead  to  the  selection  of  a 
TS,  validation  of  an  ATD  need,  and  development  of  a  plan  to 
deliver  an  ATD  in  a  timely  manner.  In  essence,  to  select 
a  TS,  perform  a  cost-benefit  analysis  (Appendix  C)  and 
make  recommendations  to  TSPO  and  user  management. 

1.3.1  REVIEW  TRAINING  SYSTEM  ASSUMPTIONS  AND  PARA¬ 
METERS  .  In  the  last  phase,  we  developed  alternative  TSs 
which  were  based  on  assumptions.  If  more  information  is 
now  available,  then  refine  these  assumptions  and  perform 
subsequent  reconfiguration  of  the  alternatives.  This  is 
necessary  since,  if  assumptions  change,  the  alternatives 
based  on  those  assumptions ,  must  also  change. 

1.3.2  DETERMINE  TRAINING  SYSTEM  COSTS.  As  each 
alternative  is  developed,  the  analyst /engineer  makes  cost 
estimates  based  on  the  information  available  on  the  W/S 
and  the  TS.  Now,  re-evaluate  the  cost  estimates  to  ensure 
that  they  reflect  the  latest  information  and  the  latest 
alternative  configuration.  In  actuality,  this  process  of 
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alternative  refinement  and  cost  estimating  is  continuous 
and  is  included  here  to  illustrate  its  logical  place  in  the 
model.  Also,  it  is  natural  to  perform  this  function  as  the 
alternatives  are  defined,  since  alternatives  with  cost  es¬ 
timates  beyond  the  funding  profiles  must  be  eliminated  at 
1.2.3. 

1.3.3  DETERMINE  TRAINING  SYSTEM  BENEFITS.  Evaluate 
the  benefits  of  the  competing  systems.  Theoretically,  the 
way  to  assess  benefit  is  to  measure  the  training  effective¬ 
ness  for  each  alternative  TS  before-the-fact.  However, 
there  is  no  universally  accepted  method  to  do  this.  A 
method  to  do  this  vague,  but  important,  function  is  to  use 
expert  opinion.  Ask  some  experts  to  rate  the  different 
alternatives  using  a  series  of  statements  and  Likert  scales. 
For  example,  the  following  scale  could  be  used: 

1  2  3  4  5 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

Sum  these  ratings  to  develop  a  measure  of  training  effec¬ 
tiveness  for  each  alternate  TS.  Other  methods  may  also  be 
used;  the  important  point  is  that  it  is  critical  that  the 
benefit  of  each  TS  be  evaluated,  documented,  and  compared. 

1.3.4  SELECT  TRAINING  SYSTEM.  Based  on  costs, 
benefits,  and  other  considerations,  management  decides 
which  TS  is  "best."  This  must  be  a  coordinated  decision 
between  the  TSPO  and  the  user  MAJCOM.  When  the  decision 
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is  made,  all  the  components  of  the  TS  are  somewhat  defined. 
Management  should  make  this  decision  as  late  as  possible  to 
enable  the  system  to  loop  between  processes  1.2  and  1.3 
with  more  than  one  iteration.  The  latest  time  this  deci¬ 
sion  can  be  made  is  the  training  system  development  sche¬ 
duled  date.  A  need  for  media  now  is  identified  and  con¬ 
sists  of  alternative  ways  to  support  the  baseline  with 
technical  orders,  films,  pictures,  ATDs,  etc.  For  each 
alternative,  the  ATD  will  support  different  tasks  and  will 
reflect  different  configurations  and/or  fidelity  require¬ 
ments.  This  development  of  alternative  media  needs,  was 
performed  in  1.2  to  the  greatest  extent  possible.  Now, 
refine  these  before  preparing  a  SON  for  validation  of  the 
need. 

1.3.5  VALIDATE  THE  NEED.  Prepare  a  SON  and  follow 
the  necessary  review  IAW  AFR  57-1  and  AFR  50-11.  After 
approval,  the  W/S  PMD  is  modified  to  show  this  new  W/S  pro¬ 
gram  component. 

1.3.6  DEVELOP  CONTRACTING  STRATEGY.  TSPO  must 
submit  a  document  to  management  that  reflects  the  appro¬ 
priate  contracting  strategy  for  the  acquisition  program. 

For  example,  if  it  is  likely  that  the  ATD  will  not  meet 
IOC,  the  W/S  and  ATD  users  must  decide  if  a  late  delivery 
is  acceptable.  If  not,  then  consider  P3I  or  actual  equip¬ 
ment  use  for  training.  The  contracting  strategy  decision 
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is  critical  and  is  included  in  this  model  for  emphasis. 

2.0  DESIGN  ATP  ELEMENT.  Figure  4-9  depicts  the 
two  logical  processes  of  this  phase — Select  ATD  Element 
(2.1)  and  Develop  Product  Specifications  (2.2).  While 
other  elements  of  the  TS 1 s  TP  are  developed  by  other  parts 
of  the  TSPO  and  user  MAJCOM,  this  phase  continues  with  the 
ATD  element  of  the  TP.  However,  continued  coordination 
with  other  TP  element  developers  is  needed  to  prevent  TS 
disjoints . 

Two  additional  points  need  be  made.  First,  as 
stated  earlier,  parts  of  this  process  are  accomplished  con¬ 
currently  with  TS  selection.  The  ATD  element  is  an  impor¬ 
tant  part  of  TS  development  and  selection,  so  when  the  SON 
is  validated,  a  set  of  ATD  systems  for  the  selected  TS  are 
already  defined  on  a  functional  level.  The  second  point 
is  that  the  ATD  cost-benefit  analysis  is  iterative.  The 
designers  and  training  developers  try  numerous  ATD  config¬ 
urations  (i.e.,  different  types,  capability,  and  mixes  of 
ATDs)  . 

2.1  SELECT  ATD  ELEMENT.  The  alternative  ATD  solu¬ 
tions  identified  in  the  SON  are  further  developed  into  more 
detailed  functional  specifications  and  a  decision  made  on 
which  element  to  develop  further.  The  functional  specifi¬ 
cations  should  be  in  enough  detail  to  permit  cost  and  bene¬ 
fit  determinations.  Figure  4-10  shows  the  four  processes 
involved  in  2.1. 
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2.1.1  DEVELOP  ALTERNATIVE  ATP  CONFIGURATIONS 


(WHICH  MEET  SON  REQUIREMENTS) .  The  TSPO  engineer  now  has 
alternative  media  approaches  and  a  project  baseline  to 
guide  the  functional  development  of  the  ATD  element.  We 
define  functional  requirements  as  things  which  the  ATD  is 
able  to  do  or  general  cues  it  is  expected  to  simulate. 

The  different  media  approaches,  funding  constraints,  tech¬ 
nological  constraints,  and  user  desires  suggest  alternative 
ATD  needs  and  configurations.  Contracted  studies  are  also 
useful  in  developing  alternatives. 

2.1.2  DETERMINE  COSTS.  For  each  ATD  system,  dev¬ 
elop  a  cost  estimate. 

2.1.3  DETERMINE  BENEFITS .  A  priori,  measure  the 
benefits  of  each  ATD  system.  A  gross  measure  at  this  point 
of  the  ATD  development  is  a  summation  of  the  C/F/D  ratings 
for  the  group  of  tasks  each  ATD  system  supports.  Further, 
look  at  the  training  effectiveness  of  each  system  in  the 
same  manner  of  module  1.3.3.  Now,  rank  the  alternatives 

in  terms  of  benefits. 

2.1.4  SELECT  ATD  ELEMENT.  The  TSPO  and  the  user 
must  look  at  the  costs  and  benefits  associated  with  each 
ATD  element  (set  of  ATDs)  and  decide  which  alternative 
gives  the  most  benefit  for  cost.  Of  course,  as  in  any 
management  decision  process,  other  considerations  play  a 
part  in  the  decision.  The  output  of  2.1.4  is  a  functional 
specification  for  the  selected  ATD  element. 
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2.2  DEVELOP  PRODUCT  SPECIFICATION.  In  this  phase. 
Figure  4-11,  the  functional  specification  becomes  a  detailed 
product  specification  for  each  ATD.  The  engineer,  using  the 
GOB's  MD,  transforms  the  functional  specification  into  the 
detail  necessary  for  a  production  specification. 

2.2.1  DEVELOP  PRODUCT  SPECIFICATION.  The  func¬ 
tional  specifications  used  today  are  transformed  into  engi¬ 
neering  oriented  product  specifications  needed  for  the  pro¬ 
duction  contract.  The  difficulty  and  accuracy  of  this  pro¬ 
cess  depends  a  great  deal  on  the  media  data  of  the  GDB . 

Early  MD  versions  require  more  engineering  interaction  and 
increase  the  chance  of  error,  while  the  ultimate  MD  version 
has  less  engineering  interaction  and  less  chance  of  error. 

2.2.2  IMPLEMENT  CONTRACTING  STRATEGY.  The  ATD 
system  specification  must  be  in  line  with  the  contracting 
strategy  in  the  ATD  Program  Plan.  For  instance,  if  P3I  is 
the  approach  that  was  adopted,  then  the  engineer  must  add 
functions  to  the  ATD  only  as  the  portion  of  the  baseline 
associated  with  that  function  is  frozen/finalized  by  the 
TSPO  and  users.  The  engineers  must  identify  the  features 
of  the  ATD  element  that  may  need  modification  later  to  meet 
these  other  requirements. 
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CHAPTER  5 


VALIDATION  WALKTHROUGH 

Chapter  5  covers  phase  seven  (Validation  Walk¬ 
through)  of  our  modified  Weinberg  methodology.  The  purpose 
of  phase  seven  was  to  subject  the  system  model  to  experts 
in  the  training  and  ATD  development  community.  The  output 
of  phase  seven  is  the  result  of  experts'  evaluations /com¬ 
parisons  between  the  current  and  proposed  training  and  ATD 
development  systems.  The  comparisons  are  based  on  the  cri¬ 
teria  stated  in  Chapter  3 ,  and  the  reader  is  reminded  that 
the  criteria  are  based  on  limitations  found  in  the  current 
system.  Section  1  reports  the  preparation  for  validation, 
and  Section  2  reports  the  results  of  the  validation. 

SECTION  1;  VALIDATION  PREPARATION 

Preparation  for  the  validation  involves  three  areas: 

1.  interviewee  selection, 

2.  statement  preparation, 

3.  initial  mathematics. 

INTERVIEWEE  SELECTION.  Mr.  Robert  E.  Coward  selected 
the  interviewees  with  one  exception.  Being  unable  to  contact 
one  of  the  selected  interviewees,  we  asked  Captain  Terrance 
Seman  of  the  AFIT  faculty  to  act  as  a  substitute.  Captain 
Seman  was  an  ATC  training  developer  for  the  P-4  fighter 
before  being  assigned  to  AFIT  and  satisfied  our  expertise 
standard.  The  six  interviewees  are: 
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1.  Captain  Terrance  R.  Seman,  AFIT  Instructor  of 
Logistics  Management,  formerly  F-4  System  Program  Manager, 
3785th  Field  Training  Group  (ATC)  ,  Sheppard  AFB  TX  (35). 

2.  Mr.  Robert  E.  Coward,  GS-13,  Special  Assistant, 
Deputy  for  Simulators,  ASD/YW  (4). 

3.  Mr.  Thomas  B.  Kelly,  GS-13,  Deputy  Branch  Chief, 
Flight  Simulation  Branch,  ASD/ENETS  (22) . 

4.  Lt  Col  Ronald  C.  Decker,  Operations  System  Mana¬ 
ger,  Detachment  4,  MACSO  (MAC)  (8). 

5.  Mr.  Thomas  W.  Hoog,  GS-13,  Electronics  Engineer, 
ASD/ENETV  (19) . 

6.  Captain  Lee  D.  Puckett,  Liaison  Research  Engi¬ 
neer,  AFHRL  (32). 

STATEMENT  PREPARATION.  Our  goal  was  to  compare  the 
current  and  our  proposed  training  and  ATD  development  sys¬ 
tems  against  each  other  using  the  six  criteria  presented  in 
Chapter  3,  Section  1  as  the  yardstick.  The  statements  and 
associated  criteria  are  presented  in  Section  2  and  were 
initially  screened  by  Dr.  Robert  Steele,  AFIT  faculty,  for 
general  correctness. 

INITIAL  MATHEMATICS.  Initial  mathematics  involves 
determining  the  statement  rejection  region.  As  noted  in 
Chapter  2,  we  are  using  a  t-statistic  to  measure  the  sta¬ 
tistical  significance  of  each  statement.  We  set  our  confi¬ 
dence  level  at  90%  with  five  (n-1)  degrees  of  freedom. 

Using  Table  II  in  Winkler  and  Hays,  we  selected  the  null 
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hypothesis  rejection  region  at  1.476  or  greater  (53:xv). 
Statements  with  t-statistics  equal  to  or  greater  than  1.476 
indicate  that  our  proposed  system  received  scores  higher 
than  the  current  system  with  90%  confidence  that  the  differ¬ 
ence  is  not  due  to  chance. 

SECTION  2:  VALIDATION  RESULTS 

For  each  criterion  we  present  the  criterion,  the 
statement  used  to  measure  that  criterion,  the  Likert  scores, 
the  t-statistic,  and  a  statement  concerning  statistical  sig¬ 
nificance  . 

CRITERION  ONE. 

Criterion .  The  system  output  must  produce  ATD  re¬ 
quirements  for  an  emerging  weapon  system  based  on  the  infor¬ 
mation  available  at  the  time  the  analysis  is  required. 

Measurement  statement.  Sufficient  information  is 
available  in  the  weapon  system  program's  conceptual  and 
validation  phases  to  develop  ATD  requirements. 

Likert  scores.  Scores  are  presented  in  Table  5-1. 


SUBJECT 

CURRENT 

SYSTEM 

PROPOSED 

Seman 

2 

4 

Coward 

2 

4 

Kelly 

2 

4 

Decker 

2 

4 

Hoog 

1 

2 

Puckett 

2 

3 

Table  5-1.  Criterion  One  Interview  Results. 
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T-statistic .  The  t-statistic  for  criterion  one  is 


7.905. 

Statistical  significance.  The  t-statistic  score  of 
7.905  is  greater  than  1.476;  therefore,  we  reject  the  null 
hypothesis. 

CRITERION  TWO. 

Criterion.  After  the  initial  iteration  of  ATD  re¬ 
quirements,  the  system  must  revise/improve  the  ATD  require¬ 
ments  as  new  information  becomes  available. 

Measurement  statement.  Periodically,  new  weapon  sys¬ 
tem  information  is  considered,  and  appropriate  adjustments  to 
the  ATD  definition  are  made. 

Likert  scores.  Scores  are  presented  in  Table  5-2. 


SUBJECT 

CURRENT 

SYSTEM 

PROPOSED 

Seman 

4 

5 

Coward 

4 

5 

Kelly 

3 

4 

Decker 

2 

4 

Hoog 

4 

4 

Puckett 

4 

4 

Table  5-2.  Criterion  Two  Interview  Results. 


T-statistic.  The  t-statistic  for  criterion  two  is 

2.71. 

Statistical  significance.  The  t-statistic  score  of 
2.71  is  greater  than  1.476;  therefore,  we  reject  the  null 
hypothesis. 
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CRITERION  THREE. 


Criterion.  The  system  must  produce  ATDs  with  only 
those  capabilities  essential  to  training  identified  tasks 
(i.e.,  prevent  "goldplating"). 

Measurement  statement.  ATD  configurations  represent 
the  minimum  capability  needed  to  satisfy  the  training  re¬ 
quirements  . 

Likert  scores.  Scores  are  presented  in  Table  5-3. 


SUBJECT 

CURRENT 

SYSTEM 

PROPOSED 

Seman 

4 

4 

Coward 

2 

4 

Kelly 

2 

4 

Decker 

3 

4 

Hoog 

4 

2 

Puckett 

2 

4 

Table  5-3.  Criterion  Three  Interview  Results. 


T-statistic.  The  t-statistic  for  criterion  three  is 

1.274. 

Statistical  significance.  The  t-statistic  score  of 
1.274  is  less  than  1.476;  therefore,  we  fail  to  reject  the 
null  hypothesis. 

CRITERION  FOUR. 

Criterion.  The  system  must  aid  ATD  acquisition  de¬ 
cision  makers  (for  example;  user,  W/S  SPO,  and  SIMSPO)  with 
cost/capability/tasks-trainable  trade-offs . 

Measurement  statement.  Two  measurement  statements 
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are  used  for  this  criterion: 

1.  The  users  and  engineers  can  quickly  assess  the 
effect  of  funding  changes  on  the  proposed  training  program. 

2.  The  users  and  engineers  can  develop  ATD  configu¬ 
ration  alternatives  that  represent  comparable  levels  of 
training  effectiveness. 

Likert  scores.  The  summation  of  the  two  scores  are 
presented  in  Table  5-4 . 


SUBJECT 

CURRENT 

SYSTEM 

PROPOSED 

Seman 

4 

8 

Coward 

5 

9 

Kelly 

4 

6 

Decker 

6 

6 

Hoog 

3 

6 

Puckett 

5 

7 

Table  5-4.  Criterion  Four  Interview  Results. 


T-statistic.  The  t-statistic  for  criterion  four  is 

4.037. 

Statistical  significance.  The  t-statistic  score  of 
4.037  is  greater  than  1.476;  therefore,  we  reject  the  null 
hypothesis . 

CRITERION  FIVE. 

Criterion.  The  system  must  improve  the  probability 
of  ATD  delivery  at  IOC  or  earlier  than  the  current  system. 

Measurement  statement.  Training  effective  ATD  can 
be  developed  and  delivered  by  weapon  system  IOC. 
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Likert  scores.  Scores  are  presented  in  Table  5-5. 


SUBJECT 

CURRENT 

SYSTEM 

PROPOSED 

Seman 

4 

3 

Coward 

2 

4 

Kelly 

2 

4 

Decker 

4 

4 

Hoog 

2 

4 

Puckett 

1 

3 

Table  5-5.  Criterion  Five  Interview  Results. 


T-statistic.  The  t-statistic  for  criterion  five  is 

2.150. 

Statistical  significance.  The  t-statistic  score  of 
2.150  is  greater  than  1.476;  therefore,  we  reject  the  null 
hypothesis. 

CRITERION  SIX. 

Criterion.  The  system  must  shorten  the  time  between 
the  start  of  W/S  analysis  and  completion  of  final  ATD  re¬ 
quirements  . 

Measurement  statement .  The  amount  of  time  it  takes 
to  identify  ATD  needs  for  new  weapon  systems  is  about  right. 
Likert  scores.  Scores  are  presented  in  Table  5-6. 
T-statistic.  The  t-statistic  for  criterion  six  is 

1.860. 

Statistical  significance.  The  t-statistic  score  of 
1.860  is  greater  than  1.476;  therefore,  we  reject  the  null 
hypothesis. 
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SUBJECT 

SYSTEM 

CURRENT  PROPOSED 

Seman 

Coward 

Kelly 

Decker 

Hoog 

Puckett 

2  4 

1  5 

2  3 

4  2 

2  4 

2  4 

Table 

5-6.  Criterion 

Six  Interview  Results. 

SUMMARY  OF  RESULTS. 

A  summary  of  the  results  for  th< 

six  criteria 

is  presented  in 

Table  5-7. 

CRITERION 

T - ST AT I ST IC 

REJECT /FAIL  TO  REJECT  NULL 

One 

7.905 

Reject 

Two 

2.71 

Reject 

Three 

1.274 

Fail  to  reject 

Four 

4.037 

Reject 

Five 

2.150 

Reject 

Six 

1.860 

Reject 

Table  5-7.  Summary  of  Interview  Results. 
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Chapter  6 


CONCLUSIONS  AND  RECOMMENDATIONS 

Chapter  6  covers  phase  eight  (Post-Evaluation)  of 
our  modified  Weinberg  methodology  by  presenting  conclusions 
(Section  1)  and  recommendations  (Section  2) . 

SECTION  1;  CONCLUSIONS. 

Our  research  objective  was  to  use  a  structured  ana¬ 
lysis  methodology  to  improve  the  process  to  identify  and  to 
communicate  ATD  requirements.  To  this  end,  we  developed  our 
system  model  (Chapter  4);  and,  using  expert  opinion,  we  com¬ 
pleted  a  validation  of  our  system  model  (Chapter  5) . 

For  five  of  the  six  criteria,  we  rejected  the  null 
hypothesis  that  our  system  was  not  an  improvement,  with 
statistical  significance  at  the  ninety  per  cent  confidence 
level.  Even  the  criterion  with  which  we  failed  to  reject 
the  null  hypothesis  had  a  mean  score  higher  for  our  system 
model  than  the  current  system.  Based  on  our  validation 
results,  we  conclude  that  there  is  a  strong  indication  that 
our  system  model  is  an  improvement  over  the  current  system. 
SECTION  2 :  RECOMMENDATIONS ♦ 

SYSTEM  MODEL.  We  recommend  that  the  model  we  have 
developed  be  implemented  by  the  United  States  Air  Force. 

We  recognize  that  our  proposed  system  is  a  significant 
departure  from  the  current  way  of  "doing  business."  How¬ 
ever,  it  incorporates  the  solution  to  many  of  the  existing 


116 


problems  with  ATD  acquisition  as  well  as  the  development  of 
training  systems ,  in  general . 

Some  aspects  of  the  model  can  be  used  right  away 
with  little  expenditure  of  resources: 

1.  Delivery  strategies/scenario  planning, 

2.  Alternative  development, 

3.  Cost/benefit  analysis, 

4.  Team  concept. 

This  is  because  these  are  really  only  changes  in  philosophy. 
Of  course,  to  implant  these  ideas  firmly  in  the  training 
system  acquisition  community  will  take  time  and  constant 
attention  by  management. 

Other  model  characteristics  will  be  more  difficult 
to  implement  because  of  resource  expenditures  and  behavioral 
considerations : 

1 .  GDB  technology, 

2.  Mission  analysis, 

3.  Centralization/collocation. 

These  will  take  more  time,  and  resource  expenditure  may  be 
high.  However,  the  benefits  to  be  derived  will  be  far- 
reaching  starting  with  initial  improvements;  and,  through 
technical  interchanges  between  team  members,  leading  to 
synergistic  improvements  in  the  future. 

AREAS  FOR  ADDITIONAL  RESEARCH.  Some  aspects  of  our 
system  model  were  based  on  ongoing  research  and  require  more 


work: 


1 .  Additional  research  is  needed  in  two  areas  of 
cuing.  Several  researchers  identified  lack  of  cuing  data 
as  the  cause  of  "goldplating"  ATDs.  Before  we  can  truly 
attempt  to  design  ATDs  with  only  essential  capabilities, 
cuing  must  be  better  understood.  First,  the  minimum  cue 
fidelity  needed  to  train  tasks  needs  development;  and 
second,  a  method  to  identify,  select,  and  standardize  the 
cues  needed  to  train  specific  tasks  needs  to  be  developed. 

2.  Although  current  GDB  research  indicates  that 
GDBs  can  be  developed,  many  questions  still  need  to  be 
answered.  First,  since  many  of  the  existing  ATD  systems 
were  developed  using  procedures  inadequate  for  developing 
weapon  systems  (ISD  task  analysis) ,  will  the  use  of  this 
data  diminish  GDB's  usefulness?  Second,  a  method  must  be 
developed  to  enable  the  GDB  development  personnel  to  iden¬ 
tify  "common  tasks."  This  is  because  the  existing  data 
give  different  names  to  like  tasks;  and,  therefore,  it  will 
be  inappropriate  to  merely  input  raw,  task  data  into  the 
GDB.  It  must  first  be  translated  or  normalized  into  a 
standardized  terminology. 

3.  Our  concept  of  media  data  is  that  it  must  indi¬ 
cate  that  the  task  in  question  (1)  can  be  trained  with 
various  types  of  media;  (2)  can  provide  a  relative  ranking 
of  which  medium  is  best  for  the  task;  and,  (3)  include  a 
cost  factor  with  each  media  alternative  that  will  indicate, 
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for  example,  that  although  a  task  can  be  most  effectively 
trained  in  a  Weapon  System  Trainer  (WST) ,  inclusion  might 
lead  to  an  unacceptable  increase  in  WST  cost.  Research  in 
these  areas  would  test  these  concepts  and  determine  the  best 
way  to  incorporate  media  data  into  the  GDB. 

4.  We  feel  that  our  system  model  is  also  applicable 
to  maintenance  training  devices  for  new  weapon  systems. 

This  is  because,  despite  the  differences  between  operations 
and  maintenance  activities,  the  ATD  acquisition  environment 
presented  in  Chapter  1  is  also  the  environment  in  which 
maintenance  devices  are  developed.  They  must  work  under  the 
same  regulatory  guidance,  with  the  same  acquisition  offices, 
and  with  the  same  time  frames.  Further,  many  of  the  problems 
we  identified  with  ATD  acquisitions  were  revealed  to  us  by 
simulator  development  personnel  who  participate  in  both 
maintenance  and  operator  training  device  programs.  There¬ 
fore,  further  research  needs  to  be  completed  to  determine 
if  our  system  model  can  be  extended  to  maintenance  training 
devices. 
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ACRONYMS  AND  ABBREVIATIONS 
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AAF :  Army  Air  Force 


AFHRL :  Air  Force  Human  Resources  Laboratory 

AFIT;  Air  Force  Institute  of  Technology 

AFM:  Air  Force  Manual 

AFP;  Air  Force  Pamphlet 

AFR:  Air  Force  Regulation 

AFSC :  Air  Force  Systems  Command 

AGARD;  Advisory  Group  for  Aerospace  Research  and  Development 

ARI :  United  States  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences 

ASP:  Aeronautical  Systems  Division 

ATC:  Air  Training  Command 

ATP:  Aircrew  Training  Device 

ATD-A:  Aircrew  Training  Device  -  Version  A 

BO:  Behavioral  Objectives 

CASDAT :  Computer  Aided  System  for  the  Development  of  Aircrew 
Training 
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C/F/D:  Criticality,  Frequency,  and  Difficulty  of  Perform' 
ance 

CFT:  Cockpit  Procedures  Trainer 
CRI:  Criterion-Referenced  Instructions 
DFDs  Data  Flow  Diagram 

; 

POD;  Department  of  Defense 

/ 

EPF:  Engineering  Design  Format 
FD:  Functional  Description 
FEA:  Front  End  Analysis 
FP:  Flight  Phase 
FS:  Functional  Statement 
FSD;  Full  Scale  Development 
GDB:  Generic  Data  Base 

GS:  Government  Service 
HQ:  Headquarters 

HumRRO;  Human  Resources  Research  Office 
ILS:  Integrated  Logistics  Support 
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INFO:  Information 


IOC:  Initial  Operational  Capability 

IRF:  Initial  Requirements  Format 

ISP:  Instructional  Systems  Development 

MAJCOM:  Major  Command 

MC:  Military  Characteristic 

MD:  Media  Data 

MGT:  Management 

MOA:  Memorandum  of  Agreement 

MT:  Mission  Trainer 

NATO:  North  Atlantic  Treaty  Organization 
NTEC:  Naval  Training  Equipment  Center 
OFT:  Operational  Flight  Trainer 
OT&E:  Operational  Test  and  Evaluation 
3 

P  I:  Pre-Planned  Product  Improvement 

PM:  Program  Manager 

PMD:  Program  Management  Directive 
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PTT:  Part  Task  Trainers 


RFP :  Request  For  Proposals 

SA:  Structured  Analysis 

SIMSPO :  Simulator  System  Program  Office 

SME:  Subject  Matter  Expert 

SON:  Statement  of  Operational  Need 

SOW:  Statement  of  Work 

SPO:  System  Program  Office 

S/R:  Stimulus  —  Response 

STREPS :  Simulator  Training  Requirements  and  Engineering 

Design  Study 

TL:  Task  List 

TMS:  Training  Management  System 

TNG:  Training 

TP:  Training  Plant 

TR:  Training  Requirements 

TS:  Integrated  Training  System 

TSA:  Training  Situation  Analysis 
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TSPO;  Training  System  Program  Office 
URF :  User  Requirements  Format 
USAF:  United  States  Air  Force 
W/S:  Weapon  System 
WST:  Weapon  System  Trainer 
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APPENDIX  B 

HOW  TO  DEVELOP  ALTERNATIVES 
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Mulligan  and  Funaro  say  that  alternative  develop¬ 
ment  is  a  creative  activity  with  "no  known  proceduraliza- 
tion  .  .  .  (of  the  activities) .  .  .  involved  in  generating 
alternative  action  plans  [29:17]."  They  suggest  that  the 
analyst  begin  by  defining  an  ideal  alternative  with  no 
constraints  considered.  Then,  the  analyst  should  introduce 
constraints  into  his  analysis  in  order  to  define  additional 
alternatives,  in  such  a  way,  as  to  account  for  all  possible 
trade-offs. 

The  DOD  Economic  Analysis  Handbook  reiterates 
the  idea  that  the  development  of  alternatives  is  an  acquired 
skill  (9:3).  Like  Mulligan  and  Funaro,  this  handbook,  pre¬ 
pared  by  the  Defense  Economic  Analysis  Council,  says  that 
it  is  the  analyst's  job  to  "study  all  feasible  alternatives 
and  to  present  to  the  decision  maker  those  alternatives  most 
cost  effective  [9:3]."  The  handbook  says  that  the  analyst 
must  select  alternatives  with  constraints  in  mind.  This 
helps  to  identify  alternatives  that  are,  in  fact,  different 
and  to  eliminate  those  that  are  infeasible.  Finally,  the 
handbook  mentions  that  the  analyst  must  also  test  the  alter¬ 
natives  under  uncertainty.  The  insights  gained  from  the 
uncertainty  testing  can  lead  to  a  new  alternative  that  will 
"provide  a  reasonably  good  hedge  against  a  range  of  the  more 
significant  uncertainties  [9:9]." 

David  Schilling,  in  discussing  his  scenario  planning 
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concept,  says  that  alternatives  should  be  based  on  alterna¬ 
tive  "futures  or  scenarios  [33:2]."  Schilling  believes  the 
following: 

A  set  of  alternative  futures  can  be  developed  simply 
by  combining  best-case  and  worst-case  scenarios  with  the 
most  probable  future  or  by  identifying  events  within  the 
planning  horizon  that  will  alter  historical  trends  [33:2]. 

The  differences  between  alternatives  should  be  developed  in 
light  of  those  future  events  that  could  change  the  course  of 
the  training  system  development.  The  DOD  Economic  Ana¬ 
lysis  Handbook  calls  these  future  occurences,  assumptions 
(9:5).  However,  while  the  handbook  talks  about  developing 
one  set  of  assumptions.  Schilling's  theory  suggests  that  a 
set  of  different  assumptions  should  be  developed  based  on 
different  "futures."  Finally,  Schilling  recommends  that  the 
alternatives  should  be  developed  with  a  maximum  of  common 
characteristics.  This  way  development  can  proceed  on  the 
common  aspects  first  and  as  additional  information  makes  the 
"futures"  more  certain,  development  can  continue. 

Figure  B-l  depicts  the  basic  model  for  developing 
alternatives.  The  analyst,  first,  develops  constraints, 
assumptions,  and  identifies  common  characteristics.  Then, 
the  analyst  should  work  with  decision  makers  and  the  train¬ 
ing  system  development  team  to  develop  a  collection  of  al¬ 
ternatives  that  are  reasonable.  With  a  GDB,  all  the  alter¬ 
natives  could  begin  with  a  common  core  of  characteristics 
and  building  onto  this  core  will  lead  to  alternatives.  As 
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additional  weapon  system  program  information  becomes  avail¬ 
able,  some  of  these  alternatives  will  become  less  likely, 
and  some  more  likely  and  more  detailed. 
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APPENDIX  C 

HOW  TO  CHOOSE  BETWEEN  ALTERNATIVES 
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Mulligan  and  Funaro  say  that  after  alternatives  have 


been  formulated,  the  next  step  is  to  estimate  effectiveness, 
cost,  and  risk  for  each  alternative  (29:17).  Then,  develop 
selection  criteria: 

...  a  list  of  statements,  preferably  quantitative, 
that  delineate  in  detail  the  maximum  acceptable  limits 
permitted  by  constraints  and  resources,  and  the  minimum 
acceptable  limits  afforded  by  system  objectives,  stand¬ 
ards,  and  specifications  [29:20]. 

To  estimate  effectiveness,  measure  or  judge  each 
alternative's  performance  against  "some  yardstick  against 
which  performance  may  be  evaluated  [29:18]."  Each  system 
alternative  should  be  judged  in  terms  of  productivity, 
reliability,  maintainability,  validity,  safety,  accuracy, 
acceptability,  availability,  security,  and  quality. 

Next,  determine  the  cost  of  each  alternative  using 
any  of  the  methods  discussed  in  the  DOD  Economic  Analysis 
Handbook  (9) .  The  handbook  discusses  the  parametric  method, 
engineering  method,  and  analogous  system  method.  Determine 
effectiveness  and  cost  for  each  alternative  in  a  consistent 
manner  in  order  to  allow  a  meaningful  comparison. 

Mulligan  and  Funaro  recommend  the  use  of  a  cost- 
effectiveness  index.  Record  the  index,  absolute  cost,  and 
absolute  effectiveness  for  each  alternative.  Thus,  if  two 
alternatives  have  equivalent  index  values,  then  look  at  the 
cost  and  effectiveness  values  to  enable  a  meaningful  compar¬ 
ison. 
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Then,  rank  each  alternative  according  to  risk. 

The  analyst  makes  a  subjective  judgement  based  on  the  un¬ 
certainties  he  knows  exist  with  each  alternative,  the  ten¬ 
tativeness  of  the  assumptions,  and  the  possibility  of 
actually  achieving  each  alternative.  The  analyst  develops 
a  rating  scale  and  assigns  a  risk  factor  to  each  alternative 

The  selection  criteria  defines  a  feasible  region  for 
the  evaluation  of  each  alternative.  Due  to  the  analyst's 
development  of  the  alternatives  in  light  of  constraints  and 
other  limiting  factors,  it  is  likely  that  the  alternatives 
will  already  be  within  the  feasible  region. 

The  DOD  Economic  Analysis  Handbook  has  a  more  direct 
equally  subjective,  approach.  First,  estimate  cost  for  each 
alternative  using  one  of  the  three  approaches  already  men¬ 
tioned.  Again,  the  same  costing  procedure  must  be  applied 
to  each  alternative  to  enable  a  meaningful  comparison.  Then 
quantify  the  benefits  of  each  alternative  (the  handbook 
explains  how  to  complete  this  difficult  step).  Finally,  the 
analyst  must  rank  each  alternative  according  to  one  of 
three  criteria: 

1.  Least  cost  for  a  given  level  of  effectiveness. 

2.  Most  effectiveness  for  a  given  cost  constraint. 

3.  Largest  ratio  of  effectiveness  to  cost  (9:8). 
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